Nov 03 23:28:27.202343 ip-10-0-138-137 systemd[1]: Starting Kubernetes Kubelet... Nov 03 23:28:27.576028 ip-10-0-138-137 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 23:28:27.576028 ip-10-0-138-137 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 23:28:27.576028 ip-10-0-138-137 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 23:28:27.576028 ip-10-0-138-137 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 23:28:27.581001 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.580911 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:28:27.581001 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.580997 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:28:27.581001 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581001 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:28:27.581001 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581005 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:28:27.581001 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581007 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:28:27.581001 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581011 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581015 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581017 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581020 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581023 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581025 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581028 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581046 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581049 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581052 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581055 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581057 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581060 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581063 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581066 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581068 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581071 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581074 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581077 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581079 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:28:27.581238 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581082 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581085 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581087 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581090 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581093 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581112 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581117 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581120 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581124 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581128 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581131 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581134 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581137 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581140 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581143 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581146 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581148 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581151 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581154 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581156 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:28:27.581691 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581159 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581161 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581164 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581166 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581169 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581172 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581174 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581177 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581182 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581186 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581189 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581192 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581195 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581197 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581200 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581203 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581205 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581208 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581212 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581215 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:28:27.582179 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581218 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581221 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581224 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581226 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581231 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581235 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581238 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581241 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581244 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581247 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581256 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581259 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581262 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581264 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581267 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581270 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581272 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581275 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581279 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:28:27.582680 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581282 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581285 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581287 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581290 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581292 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581295 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581298 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581300 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581303 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581306 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581308 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581311 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581314 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581316 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581319 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581323 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581326 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581329 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581331 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:28:27.583152 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581334 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581336 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.581339 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582168 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582177 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582180 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582183 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582187 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582190 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582192 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582195 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582198 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582201 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582206 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582210 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582213 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582217 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582219 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582222 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582225 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:28:27.583612 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582228 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582230 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582233 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582236 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582239 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582241 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582244 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582247 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582250 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582253 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582255 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582258 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582261 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582264 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582266 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582269 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582271 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582273 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582276 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582279 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:28:27.584103 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582281 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582284 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582286 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582289 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582292 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582294 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582298 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582300 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582303 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582306 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582309 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582311 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582314 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582316 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582319 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582322 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582325 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582328 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582332 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:28:27.584576 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582337 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582340 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582343 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582345 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582348 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582351 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582353 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582356 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582358 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582361 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582363 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582366 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582369 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582371 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582377 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582379 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582382 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582384 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582387 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:28:27.585013 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582390 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582393 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582396 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582398 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582401 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582404 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582406 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582409 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582412 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582414 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582417 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582420 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582422 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582425 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582427 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582430 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582432 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582435 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582437 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:28:27.585512 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582440 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582442 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582445 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582449 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582452 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582454 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582456 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582459 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582463 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582465 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582468 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582470 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.582993 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583000 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583005 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583009 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583012 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583014 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583017 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:28:27.585950 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583020 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583023 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583026 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583029 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583045 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583048 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583050 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583053 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583056 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583058 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583061 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583064 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583067 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583069 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583072 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583075 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583077 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583080 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583082 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:28:27.586441 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583085 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583088 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583092 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583095 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583097 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583100 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583103 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583105 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583108 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583112 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583114 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583117 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583119 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583122 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583125 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583127 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583130 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583132 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583135 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583137 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:28:27.586896 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583140 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583142 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583145 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583147 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583150 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583152 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583155 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583158 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583160 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583163 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583165 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583168 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583170 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583173 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583176 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583179 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583182 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583185 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583187 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583190 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:28:27.587375 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583192 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583195 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583198 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583201 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583204 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583206 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583209 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583211 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583214 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583216 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583219 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583221 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583224 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583226 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583229 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583231 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583234 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583236 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583239 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583241 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:28:27.587835 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583244 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583246 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583249 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583252 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583254 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583257 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583260 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583262 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583265 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583267 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583270 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583273 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583275 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583279 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583282 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583285 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583289 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583292 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583294 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.583297 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:28:27.588316 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584011 2570 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584020 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584026 2570 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584043 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584048 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584051 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584058 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584063 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584066 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584069 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584072 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584076 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584079 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584082 2570 flags.go:64] FLAG: --cgroup-root="" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584085 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584088 2570 flags.go:64] FLAG: --client-ca-file="" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584091 2570 flags.go:64] FLAG: --cloud-provider="external" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584094 2570 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584099 2570 flags.go:64] FLAG: --cluster-domain="" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584102 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584105 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584108 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584112 2570 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584116 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 23:28:27.588783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584119 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584124 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584128 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584131 2570 flags.go:64] FLAG: --contention-profiling="false" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584134 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584137 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584141 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584144 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584149 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584152 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584155 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584158 2570 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584162 2570 flags.go:64] FLAG: --enable-server="true" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584165 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584169 2570 flags.go:64] FLAG: --event-burst="100" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584172 2570 flags.go:64] FLAG: --event-qps="50" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584175 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584178 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584181 2570 flags.go:64] FLAG: --eviction-hard="" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584185 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584188 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584191 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584195 2570 flags.go:64] FLAG: --eviction-soft="" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584198 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584201 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 23:28:27.589341 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584204 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584207 2570 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584210 2570 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584213 2570 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584216 2570 flags.go:64] FLAG: --feature-gates="" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584220 2570 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584223 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584226 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584231 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584234 2570 flags.go:64] FLAG: --healthz-port="10248" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584237 2570 flags.go:64] FLAG: --help="false" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584241 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584244 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584248 2570 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584250 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584254 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584257 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584260 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584263 2570 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584266 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584269 2570 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584272 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584275 2570 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584278 2570 flags.go:64] FLAG: --kube-reserved="" Nov 03 23:28:27.590000 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584281 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584286 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584289 2570 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584292 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584295 2570 flags.go:64] FLAG: --lock-file="" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584298 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584300 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584304 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584310 2570 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584313 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584316 2570 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584319 2570 flags.go:64] FLAG: --logging-format="text" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584322 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584325 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584328 2570 flags.go:64] FLAG: --manifest-url="" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584331 2570 flags.go:64] FLAG: --manifest-url-header="" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584337 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584340 2570 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584344 2570 flags.go:64] FLAG: --max-pods="110" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584347 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584350 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584353 2570 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584357 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584360 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584363 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 23:28:27.590611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584366 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584373 2570 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584376 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584379 2570 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584382 2570 flags.go:64] FLAG: --pod-cidr="" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584385 2570 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584388 2570 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584391 2570 flags.go:64] FLAG: --pods-per-core="0" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584395 2570 flags.go:64] FLAG: --port="10250" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584399 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584402 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ca3b396a1af29bd1" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584405 2570 flags.go:64] FLAG: --qos-reserved="" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584408 2570 flags.go:64] FLAG: --read-only-port="10255" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584411 2570 flags.go:64] FLAG: --register-node="true" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584414 2570 flags.go:64] FLAG: --register-with-taints="" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584418 2570 flags.go:64] FLAG: --registry-burst="10" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584420 2570 flags.go:64] FLAG: --registry-qps="5" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584423 2570 flags.go:64] FLAG: --reserved-cpus="" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584426 2570 flags.go:64] FLAG: --reserved-memory="" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584430 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584433 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584436 2570 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584438 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584443 2570 flags.go:64] FLAG: --runonce="false" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584446 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584449 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584452 2570 flags.go:64] FLAG: --seccomp-default="false" Nov 03 23:28:27.591228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584455 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584458 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584461 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584464 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584467 2570 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584470 2570 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584473 2570 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584476 2570 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584479 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584482 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584485 2570 flags.go:64] FLAG: --system-cgroups="" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584488 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584493 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584497 2570 flags.go:64] FLAG: --tls-cert-file="" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584500 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584504 2570 flags.go:64] FLAG: --tls-min-version="" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584507 2570 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584510 2570 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584513 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584516 2570 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584518 2570 flags.go:64] FLAG: --v="2" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584523 2570 flags.go:64] FLAG: --version="false" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584527 2570 flags.go:64] FLAG: --vmodule="" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584531 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584534 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 23:28:27.591863 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584683 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584688 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584691 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584695 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584698 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584700 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584704 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584706 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584709 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584712 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584715 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584719 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584723 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584726 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584729 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584733 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584735 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584738 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584741 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:28:27.592537 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584744 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584747 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584750 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584753 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584756 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584758 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584761 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584763 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584766 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584769 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584771 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584774 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584777 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584779 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584782 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584785 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584787 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584790 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584792 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584795 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:28:27.592980 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584797 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584799 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584802 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584804 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584808 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584810 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584813 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584815 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584817 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584820 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584822 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584825 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584827 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584832 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584835 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584837 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584840 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584842 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584845 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584847 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:28:27.593472 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584850 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584852 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584855 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584857 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584861 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584865 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584868 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584870 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584873 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584876 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584878 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584881 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584883 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584886 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584888 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584891 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584893 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584896 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584899 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:28:27.593933 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584901 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584904 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584907 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584909 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584912 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584914 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584918 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584921 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584923 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584926 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584928 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584931 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584933 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584936 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584938 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584941 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584944 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584946 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584948 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:28:27.594425 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584951 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584954 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584956 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584959 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584961 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584964 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584966 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584969 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.584971 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.584996 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585060 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585065 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585069 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585072 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585075 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585078 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:28:27.594883 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585081 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585083 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585086 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585090 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585092 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585095 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585098 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585101 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585103 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585106 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585109 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585111 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585114 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585117 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585119 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585122 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585124 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585127 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585129 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585133 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:28:27.595290 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585136 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585138 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585141 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585144 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585146 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585149 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585151 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585154 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585157 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585160 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585163 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585166 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585168 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585171 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585173 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585177 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585180 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585182 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585185 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585187 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:28:27.595755 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585190 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585193 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585195 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585198 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585200 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585205 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585209 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585213 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585215 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585218 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585221 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585225 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585227 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585230 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585232 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585235 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585238 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585240 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585243 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:28:27.596303 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585245 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585248 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585250 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585254 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585256 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585259 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585261 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585264 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585268 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585270 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585273 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585276 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585278 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585281 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585283 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585286 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585289 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585292 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585294 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:28:27.596748 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585297 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585300 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585302 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585305 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585307 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585313 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585315 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585318 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585320 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585323 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585325 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585328 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585331 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585333 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585336 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585338 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585341 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585344 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585347 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585349 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:28:27.597207 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585352 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.585358 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.591859 2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.591877 2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.592598 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.592605 2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592665 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592671 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592674 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592677 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592679 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592682 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592685 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592688 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592690 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592693 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592695 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592698 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592701 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592703 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:28:27.597674 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592706 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592709 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592712 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592715 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592718 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592720 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592722 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592725 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592728 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592731 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592733 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592736 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592738 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592741 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592745 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592749 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592752 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592754 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592757 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592760 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:28:27.598150 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592763 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592765 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592768 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592771 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592773 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592776 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592778 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592781 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592784 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592786 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592789 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592791 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592794 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592796 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592799 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592801 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592804 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592807 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592809 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592812 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:28:27.598644 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592814 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592817 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592819 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592822 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592825 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592827 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592830 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592833 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592837 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592839 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592842 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592844 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592847 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592850 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592852 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592855 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592859 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592862 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592865 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:28:27.599183 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592868 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592871 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592873 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592876 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592878 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592881 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592884 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592886 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592889 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592893 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592897 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592900 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592903 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592906 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592909 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592913 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592915 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592918 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592922 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:28:27.599669 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592925 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592928 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592931 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592934 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592937 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592940 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592942 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592945 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592948 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592950 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592953 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592955 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592958 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.592961 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.592991 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 23:28:27.600178 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593176 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593181 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593184 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593187 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593189 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593192 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593195 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593197 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593200 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593203 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593205 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593208 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593211 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593214 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593217 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593220 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593222 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593225 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593228 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:28:27.600732 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593230 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593233 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593236 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593239 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593241 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593244 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593247 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593249 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593252 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593254 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593257 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593259 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593262 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593264 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593267 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593270 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593274 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593278 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593280 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:28:27.601371 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593283 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593286 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593289 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593291 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593294 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593296 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593299 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593301 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593304 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593306 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593309 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593311 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593314 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593316 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593319 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593321 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593324 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593327 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593329 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593332 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:28:27.601819 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593335 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593337 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593340 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593342 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593344 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593347 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593349 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593352 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593355 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593357 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593360 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593362 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593365 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593368 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593370 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593372 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593375 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593377 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593380 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:28:27.602376 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593382 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593385 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593387 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593391 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593394 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593397 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593400 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593402 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593405 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593407 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593410 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593413 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593415 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593420 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593422 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593425 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593427 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593430 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593433 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:28:27.602859 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593435 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593438 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593441 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593443 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593446 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593448 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593451 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593453 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593456 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593458 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.593482 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593516 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593520 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593523 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593526 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593529 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:28:27.603325 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593531 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593535 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593539 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593542 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593544 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593547 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593550 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593553 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593556 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593559 2570 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593563 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593565 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593568 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593570 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593573 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593575 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593578 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593581 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593583 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593588 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:28:27.603711 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593591 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593593 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593596 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593599 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593601 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593604 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593607 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593609 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593612 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593614 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593617 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593619 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593622 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593624 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593627 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593630 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593632 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593635 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593637 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:28:27.604199 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593640 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593642 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593645 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593649 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593652 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593654 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593656 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593660 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593664 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593667 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593670 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593673 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593677 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593680 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593682 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593685 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593687 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593690 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593692 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:28:27.604638 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593695 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593698 2570 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593700 2570 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593703 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593705 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593708 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593710 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593713 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593716 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593718 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593721 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593724 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593727 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593729 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593732 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593734 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593739 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593741 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593744 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593746 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:28:27.605086 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593749 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593752 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593754 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593756 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593759 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593763 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593766 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593769 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593772 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593774 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593777 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593780 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593782 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593785 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593787 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593791 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593793 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593796 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593798 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593801 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:28:27.605547 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593804 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:28:27.606006 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593806 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:28:27.606006 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:27.593809 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:28:27.606006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.593868 2570 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 23:28:27.606006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.595761 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 23:28:27.606006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.596615 2570 server.go:1014] "Starting client certificate rotation" Nov 03 23:28:27.606006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.596707 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 23:28:27.606006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.596743 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 23:28:27.606006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.598602 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 23:28:27.606006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.603660 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 23:28:27.612516 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.612499 2570 log.go:25] "Validated CRI v1 runtime API" Nov 03 23:28:27.615423 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.615409 2570 log.go:25] "Validated CRI v1 image API" Nov 03 23:28:27.616224 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.616198 2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 23:28:27.619246 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.619225 2570 fs.go:135] Filesystem UUIDs: map[1ccfce98-0105-40fd-beef-5c933ccf6c4c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d2e7a6e0-e1db-40a8-9367-4a47d372df81:/dev/nvme0n1p3] Nov 03 23:28:27.619289 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.619248 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 23:28:27.624913 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.624792 2570 manager.go:217] Machine: {Timestamp:2026-11-03 23:28:27.62328648 +0000 UTC m=+0.313075358 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103717 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e8496ce9b0f60757912438e752996 SystemUUID:ec2e8496-ce9b-0f60-7579-12438e752996 BootID:61da4c6f-96a0-4144-b92d-a9d78058494b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d7:e9:a7:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d7:e9:a7:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:ad:0b:67:a0:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 23:28:27.624913 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.624911 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 23:28:27.625083 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.625018 2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 23:28:27.628072 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.628051 2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 23:28:27.628211 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.628073 2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-137.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 23:28:27.628250 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.628213 2570 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 23:28:27.628250 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.628220 2570 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 23:28:27.628308 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.628253 2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 23:28:27.628308 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.628271 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 23:28:27.628960 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.628951 2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 23:28:27.632459 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.632448 2570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 23:28:27.632586 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.632577 2570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 23:28:27.633217 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.633208 2570 kubelet.go:493] "Attempting to sync node with API server" Nov 03 23:28:27.633257 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.633222 2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 23:28:27.633257 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.633232 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 23:28:27.633257 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.633241 2570 kubelet.go:405] "Adding apiserver pod source" Nov 03 23:28:27.633257 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.633249 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 23:28:27.633637 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.633616 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 23:28:27.636158 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.636140 2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 23:28:27.637880 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.637852 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 23:28:27.643441 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.643424 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 23:28:27.644156 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644143 2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 23:28:27.644193 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644174 2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 23:28:27.644339 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644329 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 23:28:27.644368 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644345 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 23:28:27.644368 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644352 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 23:28:27.644368 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644358 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 23:28:27.644368 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644364 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 23:28:27.644368 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644369 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 23:28:27.644986 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644978 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 23:28:27.645016 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.644990 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 23:28:27.645016 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.645000 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 23:28:27.645016 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.645006 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 23:28:27.645016 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.645015 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 23:28:27.645257 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.645025 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 23:28:27.646914 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.646901 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 23:28:27.646914 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.646914 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 23:28:27.650451 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.650432 2570 server.go:1263] "Started kubelet" Nov 03 23:28:27.650537 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.650506 2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 23:28:27.650597 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.650529 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 23:28:27.650633 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.650608 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 23:28:27.651115 ip-10-0-138-137 systemd[1]: Started Kubernetes Kubelet. Nov 03 23:28:27.651562 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.651544 2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 23:28:27.654520 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.654495 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-137.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 23:28:27.654625 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.654560 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-137.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 23:28:27.654625 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.654610 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 23:28:27.655208 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.655193 2570 server.go:317] "Adding debug handlers to kubelet server" Nov 03 23:28:27.656334 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.656312 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 23:28:27.656745 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.656730 2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 23:28:27.656877 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.656828 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:27.657413 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.657348 2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 23:28:27.657413 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657402 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 23:28:27.657413 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657413 2570 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 23:28:27.657606 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657425 2570 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 23:28:27.657606 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657495 2570 reconstruct.go:97] "Volume reconstruction finished" Nov 03 23:28:27.657606 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657503 2570 reconciler.go:29] "Reconciler: start to sync state" Nov 03 23:28:27.657783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657763 2570 factory.go:153] Registering CRI-O factory Nov 03 23:28:27.657783 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657783 2570 factory.go:223] Registration of the crio container factory successfully Nov 03 23:28:27.657895 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657832 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 23:28:27.657895 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657839 2570 factory.go:55] Registering systemd factory Nov 03 23:28:27.657895 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657844 2570 factory.go:223] Registration of the systemd container factory successfully Nov 03 23:28:27.657895 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657857 2570 factory.go:103] Registering Raw factory Nov 03 23:28:27.657895 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.657866 2570 manager.go:1199] Started watching for new ooms in manager Nov 03 23:28:27.658703 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.658687 2570 manager.go:319] Starting recovery of all containers Nov 03 23:28:27.662540 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.662515 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rpmh7" Nov 03 23:28:27.667184 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.666815 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 23:28:27.669378 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.669321 2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-137.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 23:28:27.673328 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.668920 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-137.us-east-2.compute.internal.18e4acc396e3b9df default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-137.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-137.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 23:28:27.650406879 +0000 UTC m=+0.340195755,LastTimestamp:2026-11-03 23:28:27.650406879 +0000 UTC m=+0.340195755,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-137.us-east-2.compute.internal,}" Nov 03 23:28:27.673944 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.673529 2570 manager.go:324] Recovery completed Nov 03 23:28:27.674319 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.674299 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rpmh7" Nov 03 23:28:27.677422 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.677402 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 23:28:27.677796 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.677783 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 23:28:27.679474 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.679456 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 23:28:27.679558 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.679485 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 23:28:27.679558 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.679497 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 23:28:27.680258 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.680193 2570 cpu_manager.go:226] "Starting" policy="none" Nov 03 23:28:27.680258 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.680205 2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 23:28:27.680258 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.680210 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 23:28:27.680258 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.680224 2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 23:28:27.681185 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.681169 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 23:28:27.681231 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.681203 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 23:28:27.681231 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.681219 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 23:28:27.682280 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.682259 2570 policy_none.go:50] "Start" Nov 03 23:28:27.682280 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.682282 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 23:28:27.682378 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.682291 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 23:28:27.684228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.684216 2570 policy_none.go:44] "Start" Nov 03 23:28:27.711611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.711583 2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 23:28:27.714662 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.714650 2570 manager.go:341] "Starting Device Plugin manager" Nov 03 23:28:27.714758 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.714674 2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 23:28:27.714758 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.714682 2570 server.go:87] "Starting device plugin registration server" Nov 03 23:28:27.714926 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.714915 2570 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 23:28:27.714962 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.714929 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 23:28:27.715059 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.715021 2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 23:28:27.715147 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.715127 2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 23:28:27.715147 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.715135 2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 23:28:27.715711 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.715692 2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 23:28:27.715783 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.715735 2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:27.815672 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.815643 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 23:28:27.816552 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.816536 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 23:28:27.816645 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.816566 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 23:28:27.816645 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.816580 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 23:28:27.816645 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.816629 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:27.828068 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:27.828015 2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:27.828068 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.828047 2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-137.us-east-2.compute.internal\": node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:27.853190 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.851846 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:27.952287 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:27.952258 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.052803 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.052763 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.099961 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.099910 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 23:28:28.099961 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.099935 2570 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 23:28:28.100113 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.099965 2570 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 23:28:28.100492 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.099996 2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 23:28:28.116523 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.116508 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 23:28:28.153734 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.153717 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.201254 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.201191 2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal"] Nov 03 23:28:28.201396 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.201291 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 23:28:28.203535 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.203520 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 23:28:28.203608 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.203553 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 23:28:28.203608 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.203565 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 23:28:28.205715 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.205702 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 23:28:28.205849 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.205834 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.205888 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.205865 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 23:28:28.206524 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.206505 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 23:28:28.206524 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.206522 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 23:28:28.206652 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.206534 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 23:28:28.206652 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.206543 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 23:28:28.206652 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.206547 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 23:28:28.206652 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.206554 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 23:28:28.208601 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.208585 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.208644 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.208620 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 23:28:28.209486 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.209470 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 23:28:28.209562 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.209497 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 23:28:28.209562 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.209507 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 23:28:28.236366 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.236348 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" node="ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.239177 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.239162 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" node="ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.254586 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.254567 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.261907 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.261890 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.261958 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.261915 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92a1bdd4dac917849714e10bd3b332b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"92a1bdd4dac917849714e10bd3b332b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.261958 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.261933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.355522 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.355445 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.362826 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.362800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.362883 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.362845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.362883 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.362862 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92a1bdd4dac917849714e10bd3b332b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"92a1bdd4dac917849714e10bd3b332b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.362942 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.362887 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92a1bdd4dac917849714e10bd3b332b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"92a1bdd4dac917849714e10bd3b332b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.362942 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.362801 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.362942 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.362930 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.455981 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.455945 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.539478 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.539444 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.542630 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.542395 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:28.556712 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.556696 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.596196 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.596172 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 23:28:28.596604 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.596319 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 23:28:28.648581 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.648522 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 23:28:28.657749 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.657728 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.678492 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.678455 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 23:23:28 +0000 UTC" deadline="2028-05-10 00:04:12.388608144 +0000 UTC" Nov 03 23:28:28.678492 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:28.678484 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13272h35m43.71012715s" Nov 03 23:28:28.759914 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.759889 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.860864 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.860844 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:28.961797 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:28.961727 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:29.062300 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.062261 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:28:29.106150 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.106098 2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 23:28:29.143171 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.143145 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 23:28:29.157295 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.157265 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:29.170884 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.170859 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 23:28:29.171750 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.171733 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:28:29.181480 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.181457 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 23:28:29.196526 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:29.196499 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a1bdd4dac917849714e10bd3b332b6.slice/crio-5145f3a64bc6602c4631ae64b6c7924ccbd8d44c072e136d017d1c02b774ca6a WatchSource:0}: Error finding container 5145f3a64bc6602c4631ae64b6c7924ccbd8d44c072e136d017d1c02b774ca6a: Status 404 returned error can't find the container with id 5145f3a64bc6602c4631ae64b6c7924ccbd8d44c072e136d017d1c02b774ca6a Nov 03 23:28:29.196976 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:29.196950 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d0df6b1c0ad92c712ad5146e3e385d.slice/crio-7222e28a9cd1391b3af0c5894d3f7be951ab2be85191b93cf47bda6e3bfa03e5 WatchSource:0}: Error finding container 7222e28a9cd1391b3af0c5894d3f7be951ab2be85191b93cf47bda6e3bfa03e5: Status 404 returned error can't find the container with id 7222e28a9cd1391b3af0c5894d3f7be951ab2be85191b93cf47bda6e3bfa03e5 Nov 03 23:28:29.201567 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.201553 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:28:29.479624 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.479597 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 23:28:29.634140 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.634113 2570 apiserver.go:52] "Watching apiserver" Nov 03 23:28:29.642902 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.642879 2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 23:28:29.643181 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.643164 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r","openshift-multus/multus-additional-cni-plugins-nsk4l","openshift-multus/network-metrics-daemon-chg92","openshift-network-operator/iptables-alerter-msjvc","kube-system/global-pull-secret-syncer-p7zhg","kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal","openshift-multus/multus-z2zls","openshift-network-diagnostics/network-check-target-s7hm2","openshift-cluster-node-tuning-operator/tuned-5ss7r","openshift-ovn-kubernetes/ovnkube-node-86q5p","openshift-dns/node-resolver-vsrwg","openshift-image-registry/node-ca-dbjll","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal","kube-system/konnectivity-agent-6ch5g"] Nov 03 23:28:29.647211 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.647193 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.649347 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.649327 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.651532 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.651515 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:29.651648 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.651576 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:29.651830 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.651813 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 23:28:29.651909 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.651827 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 23:28:29.651968 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.651909 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 23:28:29.651968 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.651939 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 23:28:29.652953 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.652932 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 23:28:29.653053 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.652994 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lsnsx\"" Nov 03 23:28:29.653053 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.653040 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q987z\"" Nov 03 23:28:29.653164 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.653076 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 23:28:29.653843 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.653830 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.656151 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.656132 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:29.656224 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.656205 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:29.656881 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.656865 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 23:28:29.658414 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.658395 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:29.660694 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.660679 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.662947 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.662925 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:29.663032 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.662998 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:29.665239 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.665222 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.667976 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.667959 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.670473 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670451 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.670594 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-cnibin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.670594 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670569 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-cni-binary-copy\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.670713 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670597 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-bin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.670713 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670623 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-multus-certs\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.670713 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670649 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-etc-kubernetes\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.670713 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-multus-daemon-config\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.670911 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670802 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:29.670911 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:29.671003 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e545f4e-ce4d-4a60-ba33-74207fb40c48-konnectivity-ca\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:29.671003 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.670971 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.671139 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671018 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-multus\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.671139 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671036 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-conf-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.671139 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671088 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-k8s-cni-cncf-io\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.671139 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-kubelet\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.671350 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671168 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1855ee11-b7e8-49f7-bf9d-4439ac060f20-hosts-file\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.671350 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671223 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1855ee11-b7e8-49f7-bf9d-4439ac060f20-tmp-dir\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.671350 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671255 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e545f4e-ce4d-4a60-ba33-74207fb40c48-agent-certs\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:29.671350 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671297 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc219915-7ac3-4e9b-863f-4c641a8b6540-host-slash\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.671350 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671331 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8wtn\" (UniqueName: \"kubernetes.io/projected/dc219915-7ac3-4e9b-863f-4c641a8b6540-kube-api-access-j8wtn\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.671577 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-socket-dir-parent\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.671577 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671377 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkst\" (UniqueName: \"kubernetes.io/projected/36489906-6e8f-4aa6-ae96-f872732808ed-kube-api-access-7kkst\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.671577 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-kubelet-config\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:29.671577 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671415 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-hostroot\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.671577 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671487 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-system-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.671577 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671531 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqglk\" (UniqueName: \"kubernetes.io/projected/1855ee11-b7e8-49f7-bf9d-4439ac060f20-kube-api-access-gqglk\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.671577 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671573 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdfj2\" (UniqueName: \"kubernetes.io/projected/f42ad3cc-de80-43bb-b548-dfa6c78618e5-kube-api-access-sdfj2\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:29.671816 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671604 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc219915-7ac3-4e9b-863f-4c641a8b6540-iptables-alerter-script\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.671816 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671624 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-dbus\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:29.671816 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.671654 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-netns\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.673978 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.673963 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.676176 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.676158 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 23:28:29.696746 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.696729 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vz5ck" Nov 03 23:28:29.709473 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.709457 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vz5ck" Nov 03 23:28:29.758499 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.758438 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 23:28:29.760679 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.760657 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 23:28:29.760770 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.760720 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 23:28:29.760976 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.760954 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 23:28:29.761174 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.761016 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 23:28:29.761174 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.761018 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 23:28:29.761174 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.761149 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 23:28:29.761344 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.761223 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 23:28:29.761344 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.761250 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 23:28:29.761344 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.761257 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 23:28:29.762570 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.762550 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 23:28:29.762989 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.762970 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 23:28:29.763094 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.763017 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jctss\"" Nov 03 23:28:29.763094 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.763052 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 23:28:29.763186 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.763095 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 23:28:29.763186 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.763108 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 23:28:29.763186 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.763140 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 23:28:29.763186 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.763165 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 23:28:29.763324 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.763196 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 23:28:29.763324 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.763239 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 23:28:29.764390 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.764375 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 23:28:29.764456 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.764443 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6xwn6\"" Nov 03 23:28:29.764456 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.764451 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w5s9r\"" Nov 03 23:28:29.764554 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.764444 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b74w2\"" Nov 03 23:28:29.764554 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.764453 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgkz4\"" Nov 03 23:28:29.764554 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.764466 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 23:28:29.764554 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.764487 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 23:28:29.764796 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.764778 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j9hzk\"" Nov 03 23:28:29.764890 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.764858 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dd24d\"" Nov 03 23:28:29.771974 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.771939 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-sys\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.772084 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.771992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-cni-binary-copy\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.772084 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.772029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-bin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.772209 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.772104 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysconfig\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.772384 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.772355 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-var-lib-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.772487 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.772417 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-multus-daemon-config\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.772487 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.772447 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:29.772694 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.772653 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-cni-binary-copy\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.772810 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.772739 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-bin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.772867 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.772844 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:29.772930 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.772918 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs podName:f42ad3cc-de80-43bb-b548-dfa6c78618e5 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:30.272892511 +0000 UTC m=+2.962681392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs") pod "network-metrics-daemon-chg92" (UID: "f42ad3cc-de80-43bb-b548-dfa6c78618e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:29.772977 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.772926 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-multus-daemon-config\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.773024 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.772992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:29.773105 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773026 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e545f4e-ce4d-4a60-ba33-74207fb40c48-konnectivity-ca\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:29.773105 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773043 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-tmp\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.773105 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-cnibin\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.773105 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.773271 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-multus\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.773271 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.773171 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:29.773271 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.773208 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret podName:9c784845-c509-4b51-86b8-ce9075600208 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:30.273198098 +0000 UTC m=+2.962986965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret") pod "global-pull-secret-syncer-p7zhg" (UID: "9c784845-c509-4b51-86b8-ce9075600208") : object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:29.773271 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773245 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-multus\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.773404 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-conf-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.773404 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773306 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.773479 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773405 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-conf-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.773479 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773424 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-host\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.773479 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m24tz\" (UniqueName: \"kubernetes.io/projected/34cd9f7a-0794-465b-9443-d6723c5505fb-kube-api-access-m24tz\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.773594 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773487 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fecf1197-13f8-4c4a-a878-c784e18385d6-ovn-node-metrics-cert\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.773594 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773504 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-script-lib\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.773594 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773546 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1855ee11-b7e8-49f7-bf9d-4439ac060f20-hosts-file\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.773594 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1855ee11-b7e8-49f7-bf9d-4439ac060f20-tmp-dir\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.773594 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773589 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1855ee11-b7e8-49f7-bf9d-4439ac060f20-hosts-file\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773668 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-modprobe-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctzq\" (UniqueName: \"kubernetes.io/projected/271a3854-3368-425b-9fcd-690738740bb8-kube-api-access-cctzq\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773752 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e545f4e-ce4d-4a60-ba33-74207fb40c48-konnectivity-ca\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773765 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-node-log\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc219915-7ac3-4e9b-863f-4c641a8b6540-host-slash\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773828 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1855ee11-b7e8-49f7-bf9d-4439ac060f20-tmp-dir\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wtn\" (UniqueName: \"kubernetes.io/projected/dc219915-7ac3-4e9b-863f-4c641a8b6540-kube-api-access-j8wtn\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773868 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc219915-7ac3-4e9b-863f-4c641a8b6540-host-slash\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-socket-dir-parent\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.773975 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkst\" (UniqueName: \"kubernetes.io/projected/36489906-6e8f-4aa6-ae96-f872732808ed-kube-api-access-7kkst\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.773991 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-netns\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-kubelet-config\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774095 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-socket-dir-parent\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774147 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-serviceca\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774189 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-conf\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774191 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-kubelet-config\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774271 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774299 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774333 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.774374 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774362 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-sys-fs\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-system-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqglk\" (UniqueName: \"kubernetes.io/projected/1855ee11-b7e8-49f7-bf9d-4439ac060f20-kube-api-access-gqglk\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774469 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc219915-7ac3-4e9b-863f-4c641a8b6540-iptables-alerter-script\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-dbus\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774603 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-tuned\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774620 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774641 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-systemd-units\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774644 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-system-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774704 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj5tv\" (UniqueName: \"kubernetes.io/projected/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kube-api-access-rj5tv\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774762 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-dbus\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774765 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-netns\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774797 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-netns\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774834 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-system-cni-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774883 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-slash\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774910 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-netd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774935 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gbd\" (UniqueName: \"kubernetes.io/projected/fecf1197-13f8-4c4a-a878-c784e18385d6-kube-api-access-v4gbd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.775063 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.774964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-device-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775000 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-systemd\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775022 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc219915-7ac3-4e9b-863f-4c641a8b6540-iptables-alerter-script\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-cnibin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775520 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-cnibin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775555 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-multus-certs\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775594 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-multus-certs\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775593 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-etc-kubernetes\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-host\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775643 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-etc-kubernetes\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.775663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775663 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c46g\" (UniqueName: \"kubernetes.io/projected/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-kube-api-access-5c46g\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-k8s-cni-cncf-io\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775810 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-k8s-cni-cncf-io\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-kubelet\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e545f4e-ce4d-4a60-ba33-74207fb40c48-agent-certs\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-systemd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-etc-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775914 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-ovn\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-env-overrides\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.775963 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-kubelet\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776001 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-log-socket\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776060 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-bin\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776084 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-kubelet\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-config\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776134 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-socket-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.776274 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-lib-modules\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776217 2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776235 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776293 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-hostroot\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfj2\" (UniqueName: \"kubernetes.io/projected/f42ad3cc-de80-43bb-b548-dfa6c78618e5-kube-api-access-sdfj2\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776411 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-hostroot\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776415 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-kubernetes\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776438 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-var-lib-kubelet\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776489 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-registration-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776506 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.777066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.776537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-run\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.780506 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.780487 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e545f4e-ce4d-4a60-ba33-74207fb40c48-agent-certs\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:29.790654 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.790629 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkst\" (UniqueName: \"kubernetes.io/projected/36489906-6e8f-4aa6-ae96-f872732808ed-kube-api-access-7kkst\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.790933 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.790843 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wtn\" (UniqueName: \"kubernetes.io/projected/dc219915-7ac3-4e9b-863f-4c641a8b6540-kube-api-access-j8wtn\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.790933 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.790843 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqglk\" (UniqueName: \"kubernetes.io/projected/1855ee11-b7e8-49f7-bf9d-4439ac060f20-kube-api-access-gqglk\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.792988 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.792964 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfj2\" (UniqueName: \"kubernetes.io/projected/f42ad3cc-de80-43bb-b548-dfa6c78618e5-kube-api-access-sdfj2\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:29.877704 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877646 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-kubernetes\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.877704 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-var-lib-kubelet\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.877948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877721 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.877948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-registration-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.877948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.877948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-run\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.877948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-sys\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.877948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysconfig\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.877948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877870 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-var-lib-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.877948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877920 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-tmp\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.877948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-cnibin\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-host\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.877995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24tz\" (UniqueName: \"kubernetes.io/projected/34cd9f7a-0794-465b-9443-d6723c5505fb-kube-api-access-m24tz\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fecf1197-13f8-4c4a-a878-c784e18385d6-ovn-node-metrics-cert\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878049 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-script-lib\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-modprobe-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctzq\" (UniqueName: \"kubernetes.io/projected/271a3854-3368-425b-9fcd-690738740bb8-kube-api-access-cctzq\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878147 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-node-log\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878149 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-sys\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-netns\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-serviceca\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878219 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-kubernetes\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-conf\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878273 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-var-lib-kubelet\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878307 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.883197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878315 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878359 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-sys-fs\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878366 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-registration-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878389 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-tuned\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878415 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878439 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-systemd-units\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5tv\" (UniqueName: \"kubernetes.io/projected/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kube-api-access-rj5tv\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878497 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878497 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-system-cni-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878540 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-slash\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878550 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-system-cni-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-netd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878594 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gbd\" (UniqueName: \"kubernetes.io/projected/fecf1197-13f8-4c4a-a878-c784e18385d6-kube-api-access-v4gbd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878613 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysconfig\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878619 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-device-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-systemd\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.884137 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878658 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-var-lib-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878681 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-host\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878707 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c46g\" (UniqueName: \"kubernetes.io/projected/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-kube-api-access-5c46g\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878759 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-systemd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878824 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-etc-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-ovn\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878875 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-env-overrides\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878898 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-kubelet\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-log-socket\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-bin\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-config\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-socket-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.878985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-lib-modules\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879024 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879138 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879190 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-run\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879227 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-slash\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879267 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-netd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879460 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-cnibin\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879523 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-host\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879537 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-device-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879596 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-systemd\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879637 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-host\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879871 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879927 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.879975 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-systemd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.880021 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-etc-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.880067 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-ovn\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.880567 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-env-overrides\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.880643 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-kubelet\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.880692 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-log-socket\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.880736 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-bin\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.885638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.881303 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.881320 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-config\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.881422 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-modprobe-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.881643 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-node-log\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.881691 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-netns\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.881694 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-lib-modules\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.881815 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-socket-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.881906 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-script-lib\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.881976 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-sys-fs\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.882131 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-serviceca\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.882283 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-conf\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.882310 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.882374 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-systemd-units\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.882675 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.882723 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.883083 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-tmp\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.885836 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-tuned\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.886505 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.885883 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fecf1197-13f8-4c4a-a878-c784e18385d6-ovn-node-metrics-cert\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.887086 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.886195 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.893914 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.893201 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c46g\" (UniqueName: \"kubernetes.io/projected/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-kube-api-access-5c46g\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:29.893914 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.893209 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 23:28:29.893914 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.893263 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 23:28:29.893914 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.893278 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kr6f7 for pod openshift-network-diagnostics/network-check-target-s7hm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:29.893914 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:29.893351 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7 podName:72d93d4f-2af9-44d5-b8e5-542b30760f34 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:30.393325328 +0000 UTC m=+3.083114214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kr6f7" (UniqueName: "kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7") pod "network-check-target-s7hm2" (UID: "72d93d4f-2af9-44d5-b8e5-542b30760f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:29.893914 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.893663 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gbd\" (UniqueName: \"kubernetes.io/projected/fecf1197-13f8-4c4a-a878-c784e18385d6-kube-api-access-v4gbd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:29.893914 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.893873 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24tz\" (UniqueName: \"kubernetes.io/projected/34cd9f7a-0794-465b-9443-d6723c5505fb-kube-api-access-m24tz\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.895503 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.895477 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj5tv\" (UniqueName: \"kubernetes.io/projected/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kube-api-access-rj5tv\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:29.895610 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.895586 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctzq\" (UniqueName: \"kubernetes.io/projected/271a3854-3368-425b-9fcd-690738740bb8-kube-api-access-cctzq\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:29.960355 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.960317 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z2zls" Nov 03 23:28:29.967294 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.967264 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:28:29.973744 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:29.973714 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36489906_6e8f_4aa6_ae96_f872732808ed.slice/crio-6830b700af5c1f0d22a5fe8f6f634f824c75e8953fc35ca8f7fc4c596f7853ce WatchSource:0}: Error finding container 6830b700af5c1f0d22a5fe8f6f634f824c75e8953fc35ca8f7fc4c596f7853ce: Status 404 returned error can't find the container with id 6830b700af5c1f0d22a5fe8f6f634f824c75e8953fc35ca8f7fc4c596f7853ce Nov 03 23:28:29.974845 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.974620 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:28:29.979032 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:29.979004 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1855ee11_b7e8_49f7_bf9d_4439ac060f20.slice/crio-77f4c610c96d62b403facb014d4d9d5f2feca820f4fbfad17e5b3a31bef1bdd3 WatchSource:0}: Error finding container 77f4c610c96d62b403facb014d4d9d5f2feca820f4fbfad17e5b3a31bef1bdd3: Status 404 returned error can't find the container with id 77f4c610c96d62b403facb014d4d9d5f2feca820f4fbfad17e5b3a31bef1bdd3 Nov 03 23:28:29.980386 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.980247 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:29.985500 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:29.985479 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc219915_7ac3_4e9b_863f_4c641a8b6540.slice/crio-bb7dca0071dac6810a7f1d35d443a582391df8ab813a348be5215b8cd99446fb WatchSource:0}: Error finding container bb7dca0071dac6810a7f1d35d443a582391df8ab813a348be5215b8cd99446fb: Status 404 returned error can't find the container with id bb7dca0071dac6810a7f1d35d443a582391df8ab813a348be5215b8cd99446fb Nov 03 23:28:29.988254 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:29.988186 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:28:29.999008 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:29.998981 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34cd9f7a_0794_465b_9443_d6723c5505fb.slice/crio-6b9c7948bf52995cb642e73a04151e8aba1a3b780b60c22f27dc6f920e26c969 WatchSource:0}: Error finding container 6b9c7948bf52995cb642e73a04151e8aba1a3b780b60c22f27dc6f920e26c969: Status 404 returned error can't find the container with id 6b9c7948bf52995cb642e73a04151e8aba1a3b780b60c22f27dc6f920e26c969 Nov 03 23:28:30.011354 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.010988 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:28:30.015012 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.014984 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:28:30.022777 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:30.022654 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271a3854_3368_425b_9fcd_690738740bb8.slice/crio-6aa0c4423bf56f007209bb369aab4fb4973eb8a0f0a72d99532be0b9d0cd137b WatchSource:0}: Error finding container 6aa0c4423bf56f007209bb369aab4fb4973eb8a0f0a72d99532be0b9d0cd137b: Status 404 returned error can't find the container with id 6aa0c4423bf56f007209bb369aab4fb4973eb8a0f0a72d99532be0b9d0cd137b Nov 03 23:28:30.025248 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.025007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:28:30.027216 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:30.027115 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a707ad_9256_4eb1_bf0f_eae7ab6c8d32.slice/crio-05d9ec6dd57613e3fff2b9fb708bae5276e1fe784c0366f884b9aa5c1c3cf757 WatchSource:0}: Error finding container 05d9ec6dd57613e3fff2b9fb708bae5276e1fe784c0366f884b9aa5c1c3cf757: Status 404 returned error can't find the container with id 05d9ec6dd57613e3fff2b9fb708bae5276e1fe784c0366f884b9aa5c1c3cf757 Nov 03 23:28:30.036439 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:30.036413 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceed0dfc_9a74_469f_a91d_aab8c3fa402d.slice/crio-245123bc0625af9257eaa466ddc1dda141b9828de38b07b4012547e5e1227a1b WatchSource:0}: Error finding container 245123bc0625af9257eaa466ddc1dda141b9828de38b07b4012547e5e1227a1b: Status 404 returned error can't find the container with id 245123bc0625af9257eaa466ddc1dda141b9828de38b07b4012547e5e1227a1b Nov 03 23:28:30.037318 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.037256 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:30.048587 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:28:30.048558 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecf1197_13f8_4c4a_a878_c784e18385d6.slice/crio-4bab34e7a561fe8fd6885618b4468a66449cae1d4a0ec3b7b200b388fb02aee2 WatchSource:0}: Error finding container 4bab34e7a561fe8fd6885618b4468a66449cae1d4a0ec3b7b200b388fb02aee2: Status 404 returned error can't find the container with id 4bab34e7a561fe8fd6885618b4468a66449cae1d4a0ec3b7b200b388fb02aee2 Nov 03 23:28:30.106738 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.106680 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" event={"ID":"ceed0dfc-9a74-469f-a91d-aab8c3fa402d","Type":"ContainerStarted","Data":"245123bc0625af9257eaa466ddc1dda141b9828de38b07b4012547e5e1227a1b"} Nov 03 23:28:30.108610 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.108580 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" event={"ID":"271a3854-3368-425b-9fcd-690738740bb8","Type":"ContainerStarted","Data":"6aa0c4423bf56f007209bb369aab4fb4973eb8a0f0a72d99532be0b9d0cd137b"} Nov 03 23:28:30.109901 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.109873 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ch5g" event={"ID":"1e545f4e-ce4d-4a60-ba33-74207fb40c48","Type":"ContainerStarted","Data":"f9d48ebd1c2d01314b0b19c3286be36a1a49f83fc58b8b79b5d0e5fbe2850c57"} Nov 03 23:28:30.112473 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.112397 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-msjvc" event={"ID":"dc219915-7ac3-4e9b-863f-4c641a8b6540","Type":"ContainerStarted","Data":"bb7dca0071dac6810a7f1d35d443a582391df8ab813a348be5215b8cd99446fb"} Nov 03 23:28:30.115893 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.115860 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vsrwg" event={"ID":"1855ee11-b7e8-49f7-bf9d-4439ac060f20","Type":"ContainerStarted","Data":"77f4c610c96d62b403facb014d4d9d5f2feca820f4fbfad17e5b3a31bef1bdd3"} Nov 03 23:28:30.118246 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.118220 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z2zls" event={"ID":"36489906-6e8f-4aa6-ae96-f872732808ed","Type":"ContainerStarted","Data":"6830b700af5c1f0d22a5fe8f6f634f824c75e8953fc35ca8f7fc4c596f7853ce"} Nov 03 23:28:30.119933 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.119909 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" event={"ID":"00d0df6b1c0ad92c712ad5146e3e385d","Type":"ContainerStarted","Data":"7222e28a9cd1391b3af0c5894d3f7be951ab2be85191b93cf47bda6e3bfa03e5"} Nov 03 23:28:30.121028 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.121000 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" event={"ID":"92a1bdd4dac917849714e10bd3b332b6","Type":"ContainerStarted","Data":"5145f3a64bc6602c4631ae64b6c7924ccbd8d44c072e136d017d1c02b774ca6a"} Nov 03 23:28:30.124392 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.124369 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" event={"ID":"fecf1197-13f8-4c4a-a878-c784e18385d6","Type":"ContainerStarted","Data":"4bab34e7a561fe8fd6885618b4468a66449cae1d4a0ec3b7b200b388fb02aee2"} Nov 03 23:28:30.125638 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.125609 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dbjll" event={"ID":"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32","Type":"ContainerStarted","Data":"05d9ec6dd57613e3fff2b9fb708bae5276e1fe784c0366f884b9aa5c1c3cf757"} Nov 03 23:28:30.126917 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.126891 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" event={"ID":"34cd9f7a-0794-465b-9443-d6723c5505fb","Type":"ContainerStarted","Data":"6b9c7948bf52995cb642e73a04151e8aba1a3b780b60c22f27dc6f920e26c969"} Nov 03 23:28:30.282730 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.282651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:30.282730 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.282682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:30.282928 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:30.282793 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:30.282928 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:30.282799 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:30.282928 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:30.282841 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret podName:9c784845-c509-4b51-86b8-ce9075600208 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:31.282825875 +0000 UTC m=+3.972614741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret") pod "global-pull-secret-syncer-p7zhg" (UID: "9c784845-c509-4b51-86b8-ce9075600208") : object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:30.282928 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:30.282858 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs podName:f42ad3cc-de80-43bb-b548-dfa6c78618e5 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:31.282850629 +0000 UTC m=+3.972639495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs") pod "network-metrics-daemon-chg92" (UID: "f42ad3cc-de80-43bb-b548-dfa6c78618e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:30.485436 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.484695 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:30.485436 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:30.484940 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 23:28:30.485436 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:30.484960 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 23:28:30.485436 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:30.484974 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kr6f7 for pod openshift-network-diagnostics/network-check-target-s7hm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:30.485436 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:30.485037 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7 podName:72d93d4f-2af9-44d5-b8e5-542b30760f34 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:31.485017209 +0000 UTC m=+4.174806094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr6f7" (UniqueName: "kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7") pod "network-check-target-s7hm2" (UID: "72d93d4f-2af9-44d5-b8e5-542b30760f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:30.712192 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.712009 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 23:23:30 +0000 UTC" deadline="2028-08-17 14:20:33.755823702 +0000 UTC" Nov 03 23:28:30.712192 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:30.712084 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15662h52m3.043743947s" Nov 03 23:28:31.100445 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:31.100360 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:31.100624 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:31.100508 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:31.293252 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:31.292586 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:31.293252 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:31.292635 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:31.293252 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:31.292772 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:31.293252 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:31.292824 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret podName:9c784845-c509-4b51-86b8-ce9075600208 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:33.292810543 +0000 UTC m=+5.982599422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret") pod "global-pull-secret-syncer-p7zhg" (UID: "9c784845-c509-4b51-86b8-ce9075600208") : object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:31.293252 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:31.293186 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:31.293252 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:31.293227 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs podName:f42ad3cc-de80-43bb-b548-dfa6c78618e5 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:33.293214872 +0000 UTC m=+5.983003738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs") pod "network-metrics-daemon-chg92" (UID: "f42ad3cc-de80-43bb-b548-dfa6c78618e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:31.494994 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:31.494868 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:31.495209 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:31.495046 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 23:28:31.495209 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:31.495085 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 23:28:31.495209 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:31.495100 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kr6f7 for pod openshift-network-diagnostics/network-check-target-s7hm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:31.495209 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:31.495167 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7 podName:72d93d4f-2af9-44d5-b8e5-542b30760f34 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:33.495148125 +0000 UTC m=+6.184936999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr6f7" (UniqueName: "kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7") pod "network-check-target-s7hm2" (UID: "72d93d4f-2af9-44d5-b8e5-542b30760f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:31.712855 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:31.712727 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 23:23:30 +0000 UTC" deadline="2028-05-20 12:15:38.250392054 +0000 UTC" Nov 03 23:28:31.712855 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:31.712824 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13524h47m6.537576802s" Nov 03 23:28:32.103195 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:32.102271 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:32.103195 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:32.102457 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:32.103195 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:32.102601 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:32.103195 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:32.102709 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:33.101115 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:33.100616 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:33.101115 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:33.100965 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:33.163575 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:33.163513 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" event={"ID":"92a1bdd4dac917849714e10bd3b332b6","Type":"ContainerStarted","Data":"553d027fb11bb3f32b2932deb9986ccd09d5ca0b8f307a519c8641a5ae50943d"} Nov 03 23:28:33.182888 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:33.182828 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" podStartSLOduration=3.18280983 podStartE2EDuration="3.18280983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:33.181604697 +0000 UTC m=+5.871393590" watchObservedRunningTime="2026-11-03 23:28:33.18280983 +0000 UTC m=+5.872598722" Nov 03 23:28:33.316358 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:33.316310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:33.316544 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:33.316370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:33.316621 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:33.316561 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:33.316688 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:33.316625 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret podName:9c784845-c509-4b51-86b8-ce9075600208 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:37.316607473 +0000 UTC m=+10.006396342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret") pod "global-pull-secret-syncer-p7zhg" (UID: "9c784845-c509-4b51-86b8-ce9075600208") : object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:33.317080 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:33.317054 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:33.317186 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:33.317134 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs podName:f42ad3cc-de80-43bb-b548-dfa6c78618e5 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:37.317101406 +0000 UTC m=+10.006890277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs") pod "network-metrics-daemon-chg92" (UID: "f42ad3cc-de80-43bb-b548-dfa6c78618e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:33.517715 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:33.517676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:33.517899 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:33.517829 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 23:28:33.517899 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:33.517843 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 23:28:33.517899 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:33.517853 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kr6f7 for pod openshift-network-diagnostics/network-check-target-s7hm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:33.517899 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:33.517899 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7 podName:72d93d4f-2af9-44d5-b8e5-542b30760f34 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:37.517886175 +0000 UTC m=+10.207675055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr6f7" (UniqueName: "kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7") pod "network-check-target-s7hm2" (UID: "72d93d4f-2af9-44d5-b8e5-542b30760f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:34.102687 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:34.102650 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:34.103194 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:34.102791 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:34.103384 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:34.103360 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:34.103524 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:34.103500 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:34.175140 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:34.173889 2570 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="ff90458e67899d62c6943f827cd60559b3946c5dad44e63faaa621bf7020d49b" exitCode=0 Nov 03 23:28:34.175140 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:34.173987 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" event={"ID":"271a3854-3368-425b-9fcd-690738740bb8","Type":"ContainerDied","Data":"ff90458e67899d62c6943f827cd60559b3946c5dad44e63faaa621bf7020d49b"} Nov 03 23:28:34.182338 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:34.182305 2570 generic.go:356] "Generic (PLEG): container finished" podID="00d0df6b1c0ad92c712ad5146e3e385d" containerID="c65a1184f38ee2ce849f8f38c3d5380241963f9395c6af9511bb6cb068a8b9c4" exitCode=0 Nov 03 23:28:34.183563 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:34.183537 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" event={"ID":"00d0df6b1c0ad92c712ad5146e3e385d","Type":"ContainerDied","Data":"c65a1184f38ee2ce849f8f38c3d5380241963f9395c6af9511bb6cb068a8b9c4"} Nov 03 23:28:35.100670 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:35.100631 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:35.100858 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:35.100790 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:35.188196 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:35.188153 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ch5g" event={"ID":"1e545f4e-ce4d-4a60-ba33-74207fb40c48","Type":"ContainerStarted","Data":"47f519ea711dc8f549e599f884b551cc0913f675dcdb39a5444b44c31f412867"} Nov 03 23:28:35.195478 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:35.195446 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" event={"ID":"00d0df6b1c0ad92c712ad5146e3e385d","Type":"ContainerStarted","Data":"974a5e39153c185fb6dd7030966914640260ecc08ff1d0b207be77ab6c404e69"} Nov 03 23:28:35.209001 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:35.207966 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6ch5g" podStartSLOduration=3.055494252 podStartE2EDuration="7.207946762s" totalImagesPullingTime="4.15245251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:35.207452727 +0000 UTC m=+7.897241617" watchObservedRunningTime="2026-11-03 23:28:35.207946762 +0000 UTC m=+7.897735654" Nov 03 23:28:36.102313 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:36.102239 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:36.102476 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:36.102393 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:36.102906 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:36.102885 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:36.103034 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:36.103005 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:36.203689 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:36.203639 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dbjll" event={"ID":"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32","Type":"ContainerStarted","Data":"d0411588ab644a56a4abb1b4da18f5fc98d2cc648aef5e0d21648f1dec949f2c"} Nov 03 23:28:36.222029 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:36.221974 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" podStartSLOduration=6.221953636 podStartE2EDuration="6.221953636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:35.227874616 +0000 UTC m=+7.917663508" watchObservedRunningTime="2026-11-03 23:28:36.221953636 +0000 UTC m=+8.911742527" Nov 03 23:28:37.100408 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:37.100327 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:37.100582 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:37.100459 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:37.209549 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:37.209511 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" event={"ID":"ceed0dfc-9a74-469f-a91d-aab8c3fa402d","Type":"ContainerStarted","Data":"9678c18aff731e4fbc25fd55601bb30b6b5451446459560b3b446b2cb66b44ed"} Nov 03 23:28:37.353073 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:37.352946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:37.353073 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:37.353001 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:37.353340 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:37.353133 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:37.353340 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:37.353182 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:37.353340 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:37.353218 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs podName:f42ad3cc-de80-43bb-b548-dfa6c78618e5 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.353198822 +0000 UTC m=+18.042987691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs") pod "network-metrics-daemon-chg92" (UID: "f42ad3cc-de80-43bb-b548-dfa6c78618e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:37.353340 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:37.353246 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret podName:9c784845-c509-4b51-86b8-ce9075600208 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.35322729 +0000 UTC m=+18.043016159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret") pod "global-pull-secret-syncer-p7zhg" (UID: "9c784845-c509-4b51-86b8-ce9075600208") : object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:37.555659 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:37.554915 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:37.555659 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:37.555179 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 23:28:37.555659 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:37.555205 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 23:28:37.555659 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:37.555219 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kr6f7 for pod openshift-network-diagnostics/network-check-target-s7hm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:37.555659 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:37.555282 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7 podName:72d93d4f-2af9-44d5-b8e5-542b30760f34 nodeName:}" failed. No retries permitted until 2026-11-03 23:28:45.555262831 +0000 UTC m=+18.245051712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr6f7" (UniqueName: "kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7") pod "network-check-target-s7hm2" (UID: "72d93d4f-2af9-44d5-b8e5-542b30760f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:38.102139 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:38.101240 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:38.102139 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:38.101394 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:38.102139 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:38.101902 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:38.102139 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:38.102001 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:38.465071 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:38.465046 2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 23:28:38.727730 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:38.727615 2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T23:28:38.46506802Z","UUID":"1b419795-6dbb-4ab9-baa6-880e1961d9a2","Handler":null,"Name":"","Endpoint":""} Nov 03 23:28:38.732086 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:38.731683 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 23:28:38.732086 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:38.731714 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 23:28:39.101156 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.101115 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:39.101352 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:39.101272 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:39.163846 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.163803 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:39.164647 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.164609 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:39.184478 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.184422 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dbjll" podStartSLOduration=5.406216198 podStartE2EDuration="11.184403329s" totalImagesPullingTime="5.778187131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:36.222401563 +0000 UTC m=+8.912190453" watchObservedRunningTime="2026-11-03 23:28:39.184403329 +0000 UTC m=+11.874192218" Nov 03 23:28:39.218326 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.218238 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" event={"ID":"ceed0dfc-9a74-469f-a91d-aab8c3fa402d","Type":"ContainerStarted","Data":"8389ff2f0fe3ea92f6e4333825ff209b2024fae3750148788c735bee6e64cf18"} Nov 03 23:28:39.221984 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.221958 2570 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="eed9ffafda1c82c8d3fa69d27e5a96518419c1a78600a06b9fc0a1a353434124" exitCode=0 Nov 03 23:28:39.222126 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.222038 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" event={"ID":"271a3854-3368-425b-9fcd-690738740bb8","Type":"ContainerDied","Data":"eed9ffafda1c82c8d3fa69d27e5a96518419c1a78600a06b9fc0a1a353434124"} Nov 03 23:28:39.225403 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.224834 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vsrwg" event={"ID":"1855ee11-b7e8-49f7-bf9d-4439ac060f20","Type":"ContainerStarted","Data":"522aa25d42252411d3ca2cb26c1a1ed904dc7b7d88cd2b563b5f1399a8433686"} Nov 03 23:28:39.227525 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.227501 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z2zls" event={"ID":"36489906-6e8f-4aa6-ae96-f872732808ed","Type":"ContainerStarted","Data":"ec874e35c7171876321c040182819e651d7bcb887d9658fd21f644ef38f44791"} Nov 03 23:28:39.298608 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:39.298538 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-z2zls" podStartSLOduration=2.692603599 podStartE2EDuration="11.298520663s" totalImagesPullingTime="8.605917064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:39.298426566 +0000 UTC m=+11.988215459" watchObservedRunningTime="2026-11-03 23:28:39.298520663 +0000 UTC m=+11.988309554" Nov 03 23:28:40.102723 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:40.102500 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:40.103148 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:40.102850 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:40.103253 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:40.103229 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:40.103365 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:40.103343 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:40.232395 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:40.232316 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" event={"ID":"ceed0dfc-9a74-469f-a91d-aab8c3fa402d","Type":"ContainerStarted","Data":"64c114bfd2d10a40a0fb809a8b808b2e6a8db426b6ad411f436888bc26efa9e4"} Nov 03 23:28:40.235037 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:40.235009 2570 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="b7cbd2508f9fa90161effcc922c0a88cc843d0af48c350302b22c9e3a4ff8a6e" exitCode=0 Nov 03 23:28:40.235172 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:40.235082 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" event={"ID":"271a3854-3368-425b-9fcd-690738740bb8","Type":"ContainerDied","Data":"b7cbd2508f9fa90161effcc922c0a88cc843d0af48c350302b22c9e3a4ff8a6e"} Nov 03 23:28:40.237910 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:40.237625 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-msjvc" event={"ID":"dc219915-7ac3-4e9b-863f-4c641a8b6540","Type":"ContainerStarted","Data":"1f1ee405db917278514e3b4da816b6a69682e8250b1d3b6bb89eefafbadd5bb5"} Nov 03 23:28:40.255794 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:40.255739 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vsrwg" podStartSLOduration=3.522514494 podStartE2EDuration="12.255721274s" totalImagesPullingTime="8.73320678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:39.319933789 +0000 UTC m=+12.009722680" watchObservedRunningTime="2026-11-03 23:28:40.255721274 +0000 UTC m=+12.945510167" Nov 03 23:28:40.256027 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:40.255992 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" podStartSLOduration=3.19852843 podStartE2EDuration="12.255980711s" totalImagesPullingTime="9.057452281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:40.255159599 +0000 UTC m=+12.944948488" watchObservedRunningTime="2026-11-03 23:28:40.255980711 +0000 UTC m=+12.945769601" Nov 03 23:28:40.321655 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:40.321588 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-msjvc" podStartSLOduration=3.716350656 podStartE2EDuration="12.321571302s" totalImagesPullingTime="8.605220646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:40.32092297 +0000 UTC m=+13.010711866" watchObservedRunningTime="2026-11-03 23:28:40.321571302 +0000 UTC m=+13.011360192" Nov 03 23:28:41.100729 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:41.100702 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:41.100845 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:41.100823 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:41.242404 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:41.242355 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" event={"ID":"34cd9f7a-0794-465b-9443-d6723c5505fb","Type":"ContainerStarted","Data":"18533a3a3f90a3b1c6f07a778bb811f34370548ccdcb53801545d0e211b5a8b9"} Nov 03 23:28:41.263633 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:41.263583 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" podStartSLOduration=2.246938794 podStartE2EDuration="13.263569751s" totalImagesPullingTime="11.016630957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:41.263213724 +0000 UTC m=+13.953002607" watchObservedRunningTime="2026-11-03 23:28:41.263569751 +0000 UTC m=+13.953358637" Nov 03 23:28:42.101108 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:42.101079 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:42.101290 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:42.101223 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:42.101350 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:42.101209 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:42.101350 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:42.101328 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:42.247004 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:42.246969 2570 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="03aedc760b9401865622845edc19c6cc0ab056306a9dff957d25132187594252" exitCode=0 Nov 03 23:28:42.247601 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:42.247065 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" event={"ID":"271a3854-3368-425b-9fcd-690738740bb8","Type":"ContainerDied","Data":"03aedc760b9401865622845edc19c6cc0ab056306a9dff957d25132187594252"} Nov 03 23:28:43.100732 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:43.100683 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:43.100919 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:43.100822 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:43.685177 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:43.685140 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:43.863949 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:43.863912 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:43.965653 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:43.965625 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:44.100836 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:44.100629 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:44.100982 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:44.100949 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:44.101230 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:44.100711 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:44.101230 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:44.101189 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:44.255049 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:44.254928 2570 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="3039918425f4a1ef0d62a257e5ea07756f0318a2f3676ffc1f5ad65963ebcc3d" exitCode=0 Nov 03 23:28:44.255049 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:44.254992 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" event={"ID":"271a3854-3368-425b-9fcd-690738740bb8","Type":"ContainerDied","Data":"3039918425f4a1ef0d62a257e5ea07756f0318a2f3676ffc1f5ad65963ebcc3d"} Nov 03 23:28:44.387458 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:44.387428 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:44.527352 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:44.527218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:44.897406 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:44.897319 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:45.003100 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:45.003067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:45.100356 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:45.100324 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:45.100489 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:45.100427 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:45.260017 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:45.259935 2570 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="b28226f4661c7e4c5b3327f04509e38c654a9388ff2a3cae2a2fc865d3a778b3" exitCode=0 Nov 03 23:28:45.260017 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:45.260000 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" event={"ID":"271a3854-3368-425b-9fcd-690738740bb8","Type":"ContainerDied","Data":"b28226f4661c7e4c5b3327f04509e38c654a9388ff2a3cae2a2fc865d3a778b3"} Nov 03 23:28:45.418948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:45.418931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:45.419016 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:45.418960 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:45.419101 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:45.419087 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:45.419150 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:45.419095 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:45.419150 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:45.419141 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret podName:9c784845-c509-4b51-86b8-ce9075600208 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:01.419126021 +0000 UTC m=+34.108914891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret") pod "global-pull-secret-syncer-p7zhg" (UID: "9c784845-c509-4b51-86b8-ce9075600208") : object "kube-system"/"original-pull-secret" not registered Nov 03 23:28:45.419227 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:45.419154 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs podName:f42ad3cc-de80-43bb-b548-dfa6c78618e5 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:01.419148216 +0000 UTC m=+34.108937080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs") pod "network-metrics-daemon-chg92" (UID: "f42ad3cc-de80-43bb-b548-dfa6c78618e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:28:45.621299 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:45.621129 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:45.621299 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:45.621292 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 23:28:45.621459 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:45.621313 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 23:28:45.621459 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:45.621322 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kr6f7 for pod openshift-network-diagnostics/network-check-target-s7hm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:45.621459 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:45.621370 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7 podName:72d93d4f-2af9-44d5-b8e5-542b30760f34 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:01.62135607 +0000 UTC m=+34.311144947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr6f7" (UniqueName: "kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7") pod "network-check-target-s7hm2" (UID: "72d93d4f-2af9-44d5-b8e5-542b30760f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:28:46.100301 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.100270 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:46.100701 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:46.100469 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:46.100701 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.100513 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:46.100701 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:46.100602 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:46.264795 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.264753 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" event={"ID":"271a3854-3368-425b-9fcd-690738740bb8","Type":"ContainerStarted","Data":"502d54d0485a9f988e5edd08b2d19c6801bf3d06d700ffa7a01889740453316b"} Nov 03 23:28:46.267437 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.267413 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" event={"ID":"fecf1197-13f8-4c4a-a878-c784e18385d6","Type":"ContainerStarted","Data":"61551c7174a829c758083f319332b487cb4be8ac09096fe7510fd1307b308280"} Nov 03 23:28:46.267537 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.267468 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" event={"ID":"fecf1197-13f8-4c4a-a878-c784e18385d6","Type":"ContainerStarted","Data":"0c7b2d36e52faede3579cd1beef03a357c47a03669c70ada7c435a70835f1fb8"} Nov 03 23:28:46.267537 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.267503 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" event={"ID":"fecf1197-13f8-4c4a-a878-c784e18385d6","Type":"ContainerStarted","Data":"13d6bd7b366e2a762c58dcbabbfd5f8f6b317e66c8ef27f4b06f91a0fcc54a80"} Nov 03 23:28:46.267537 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.267532 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" event={"ID":"fecf1197-13f8-4c4a-a878-c784e18385d6","Type":"ContainerStarted","Data":"bec818522a93abfcb3058b2b6789230c4dd57834fc8ef6a645221eb869f1e68e"} Nov 03 23:28:46.267617 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.267565 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" event={"ID":"fecf1197-13f8-4c4a-a878-c784e18385d6","Type":"ContainerStarted","Data":"5efd95241a537212698fe2982b9c4380b6ba4272131784b94906bc8d20efdbe2"} Nov 03 23:28:46.267617 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.267591 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" event={"ID":"fecf1197-13f8-4c4a-a878-c784e18385d6","Type":"ContainerStarted","Data":"959da9a062be7028f570bd1ab0fc8360ac09083ddbc38bdd4d100670aefb3208"} Nov 03 23:28:46.294145 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.294101 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nsk4l" podStartSLOduration=7.051457586 podStartE2EDuration="18.29408964s" totalImagesPullingTime="11.242632054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:46.293915615 +0000 UTC m=+18.983704502" watchObservedRunningTime="2026-11-03 23:28:46.29408964 +0000 UTC m=+18.983878519" Nov 03 23:28:46.477136 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.477058 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:46.575228 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.575196 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:46.868857 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:46.868779 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:47.100757 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:47.100725 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:47.101136 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:47.100827 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:47.911271 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:47.911175 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:48.011675 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:48.011642 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:48.103941 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:48.103592 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:48.104449 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:48.104284 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:48.104449 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:48.104315 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:48.104449 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:48.104365 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:48.274208 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:48.274129 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" event={"ID":"fecf1197-13f8-4c4a-a878-c784e18385d6","Type":"ContainerStarted","Data":"50586995dce1daba768f8f9c1854017a46a16de516bc0fa0faf6d4bd79ff0e35"} Nov 03 23:28:49.100663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:49.100635 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:49.100801 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:49.100724 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:49.123794 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:49.123759 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:49.248229 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:49.248200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:49.784481 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:49.784452 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:49.909733 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:49.909696 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:50.100857 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.100787 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:50.100857 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.100791 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:50.101016 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:50.100985 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:50.101153 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:50.101127 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:50.126493 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.126467 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:50.256442 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.256419 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:50.282077 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.282046 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" event={"ID":"fecf1197-13f8-4c4a-a878-c784e18385d6","Type":"ContainerStarted","Data":"8564c729b546e96e60113897f1491c0ead2c78d1cee00a5f653c58a3de94f95c"} Nov 03 23:28:50.282344 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.282320 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:50.297028 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.297008 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:50.318175 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.318126 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" podStartSLOduration=7.074701126 podStartE2EDuration="22.318115297s" totalImagesPullingTime="15.243414171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:28:50.31617688 +0000 UTC m=+23.005965767" watchObservedRunningTime="2026-11-03 23:28:50.318115297 +0000 UTC m=+23.007904182" Nov 03 23:28:50.362915 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.362854 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:50.722622 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.722460 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:50.976316 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:50.976214 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:51.100366 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.100338 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:51.100516 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:51.100432 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:51.117876 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.117842 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:51.283776 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.283700 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 23:28:51.284183 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.284068 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:51.292949 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.292922 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:51.299275 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.299254 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:51.404705 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.404666 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:51.611813 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.611532 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:51.714420 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.714392 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p7zhg"] Nov 03 23:28:51.714558 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.714493 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:51.714599 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:51.714568 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:51.717717 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.717690 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s7hm2"] Nov 03 23:28:51.717847 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.717833 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:51.717963 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:51.717947 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:51.718247 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.718229 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-chg92"] Nov 03 23:28:51.718503 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:51.718325 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:51.718758 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:51.718423 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:52.251222 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:52.251183 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:52.285667 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:52.285648 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 23:28:52.637659 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:52.637578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:52.782633 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:52.782601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:52.882926 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:52.882897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:53.100337 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:53.100254 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:53.100337 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:53.100284 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:53.100546 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:53.100360 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:53.100546 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:53.100407 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:53.100546 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:53.100487 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:53.100546 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:53.100533 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:53.287138 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:53.287112 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 23:28:53.419306 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:53.419240 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vsrwg_1855ee11-b7e8-49f7-bf9d-4439ac060f20/dns-node-resolver/0.log" Nov 03 23:28:53.973149 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:53.973080 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:54.166260 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.166173 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:54.174091 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.174070 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:28:54.184985 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.184946 2570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" podUID="fecf1197-13f8-4c4a-a878-c784e18385d6" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 23:28:54.193233 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.193211 2570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" podUID="fecf1197-13f8-4c4a-a878-c784e18385d6" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 23:28:54.298497 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.298463 2570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" podUID="fecf1197-13f8-4c4a-a878-c784e18385d6" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 23:28:54.424860 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.424801 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dbjll_58a707ad-9256-4eb1-bf0f-eae7ab6c8d32/node-ca/0.log" Nov 03 23:28:54.431339 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.431315 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:54.569829 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.569800 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:54.774426 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.774339 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:54.875377 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:54.875343 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:55.101155 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:55.101091 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:55.101155 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:55.101131 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:55.101315 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:55.101102 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:55.101315 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:55.101199 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:55.101315 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:55.101267 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:55.101402 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:55.101362 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:55.115284 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:55.115261 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:55.969303 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:55.969270 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:56.185597 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:56.185515 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:56.379459 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:56.379420 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:56.479086 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:56.478997 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:56.579920 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:56.579894 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:56.785154 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:56.785077 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:56.981611 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:56.981554 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:57.100500 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.100414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:57.100649 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.100414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:57.100649 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:57.100515 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:57.100649 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.100522 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:57.100649 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:57.100594 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:57.100796 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:57.100650 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:57.104514 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.104480 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:57.216407 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.216371 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:57.378243 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.378159 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:57.482256 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.482219 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:57.583934 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.583910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:57.691586 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.691510 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:57.967566 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:57.967488 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:58.107001 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.106965 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:58.244362 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.244288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:58.348861 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.348827 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:58.369764 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.369734 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:58.369893 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.369853 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 23:28:58.370338 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.370321 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:28:58.442509 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.442474 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:58.567225 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.567140 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:58.803954 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.803929 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:58.995405 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:58.995328 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:59.100929 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:59.100898 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:28:59.101118 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:59.100965 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:28:59.101118 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:59.100990 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:28:59.101118 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:59.101110 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:28:59.101277 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:59.101172 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:28:59.101277 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:28:59.101246 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:28:59.111340 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:59.111313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:59.245483 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:59.245402 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:59.406072 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:59.405997 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:59.512794 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:59.512724 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:59.985132 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:28:59.985049 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:00.172357 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:00.172319 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:00.384697 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:00.384617 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:00.583623 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:00.583596 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:00.766682 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:00.766599 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:00.935676 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:00.935639 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:01.100846 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:01.100772 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:01.101089 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:01.100772 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:01.101089 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.100881 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:29:01.101089 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:01.100891 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:01.101089 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.100957 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:29:01.101089 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.101016 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:29:01.444664 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:01.444641 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:01.445025 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:01.444672 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:01.445025 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.444776 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 23:29:01.445025 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.444802 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:29:01.445025 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.444824 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret podName:9c784845-c509-4b51-86b8-ce9075600208 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:33.444810942 +0000 UTC m=+66.134599807 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret") pod "global-pull-secret-syncer-p7zhg" (UID: "9c784845-c509-4b51-86b8-ce9075600208") : object "kube-system"/"original-pull-secret" not registered Nov 03 23:29:01.445025 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.444860 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs podName:f42ad3cc-de80-43bb-b548-dfa6c78618e5 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:33.444843878 +0000 UTC m=+66.134632758 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs") pod "network-metrics-daemon-chg92" (UID: "f42ad3cc-de80-43bb-b548-dfa6c78618e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 23:29:01.645675 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:01.645644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:01.645843 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.645763 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 23:29:01.645843 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.645778 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 23:29:01.645843 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.645787 2570 projected.go:196] Error preparing data for projected volume kube-api-access-kr6f7 for pod openshift-network-diagnostics/network-check-target-s7hm2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:29:01.645843 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:01.645831 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7 podName:72d93d4f-2af9-44d5-b8e5-542b30760f34 nodeName:}" failed. No retries permitted until 2026-11-03 23:29:33.645819499 +0000 UTC m=+66.335608364 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kr6f7" (UniqueName: "kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7") pod "network-check-target-s7hm2" (UID: "72d93d4f-2af9-44d5-b8e5-542b30760f34") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 23:29:02.000784 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:02.000752 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:02.167027 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:02.166947 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:02.370167 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:02.370132 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:02.565185 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:02.565104 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:02.767093 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:02.767056 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:03.100694 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:03.100664 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:03.100893 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:03.100763 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:03.100893 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:03.100766 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:29:03.100893 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:03.100882 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:29:03.101101 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:03.100898 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:03.101101 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:03.100947 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:29:03.569475 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:03.569386 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:03.768436 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:03.768396 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:03.967880 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:03.967791 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:04.116585 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:04.116547 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:04.223978 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:04.223906 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:05.100897 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:05.100869 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:05.101397 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:05.100955 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:05.101397 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:05.100967 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:05.101397 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:05.101049 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:29:05.101397 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:05.101129 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:29:05.101397 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:05.101232 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:29:06.520657 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:06.520622 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:06.655149 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:06.655113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:07.101023 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:07.100995 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:07.101241 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:07.101117 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:07.101241 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:07.101130 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:29:07.101241 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:07.101144 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:07.101241 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:07.101204 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:29:07.101447 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:07.101267 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:29:09.100821 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:09.100787 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:09.101248 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:09.100827 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:09.101248 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:09.100889 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:09.101248 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:09.100905 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:29:09.101248 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:09.100952 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:29:09.101248 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:09.101026 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:29:10.710187 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:10.710157 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:10.810312 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:10.810277 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:11.101199 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:11.101089 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:11.101383 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:11.101090 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:11.101383 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:11.101226 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:29:11.101383 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:11.101303 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:29:11.101383 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:11.101113 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:11.101383 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:11.101374 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:29:12.233015 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:12.232984 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:12.377767 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:12.377721 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:12.568273 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:12.568197 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:12.684238 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:12.684199 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:12.796470 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:12.796439 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:13.100937 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.100901 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:13.101112 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.100904 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:13.101112 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:13.101002 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7zhg" podUID="9c784845-c509-4b51-86b8-ce9075600208" Nov 03 23:29:13.101252 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:13.101119 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chg92" podUID="f42ad3cc-de80-43bb-b548-dfa6c78618e5" Nov 03 23:29:13.101252 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.100912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:13.101252 ip-10-0-138-137 kubenswrapper[2570]: E1103 23:29:13.101215 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7hm2" podUID="72d93d4f-2af9-44d5-b8e5-542b30760f34" Nov 03 23:29:13.153776 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.153746 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeReady" Nov 03 23:29:13.153926 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.153875 2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 23:29:13.219966 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.219935 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pqtnl"] Nov 03 23:29:13.221445 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.221428 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:13.221898 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.221878 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h4spw"] Nov 03 23:29:13.223362 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.223347 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.226402 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.226381 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ht52\" (UniqueName: \"kubernetes.io/projected/17b0bd49-65e6-4393-88ba-b578c5eab313-kube-api-access-9ht52\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:13.226494 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.226408 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc833b9-89ed-4f1a-8dec-602fffd9c072-config-volume\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.226494 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.226427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc833b9-89ed-4f1a-8dec-602fffd9c072-metrics-tls\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.226494 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.226442 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b0bd49-65e6-4393-88ba-b578c5eab313-cert\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:13.226603 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.226563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt84d\" (UniqueName: \"kubernetes.io/projected/2bc833b9-89ed-4f1a-8dec-602fffd9c072-kube-api-access-pt84d\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.226676 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.226661 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc833b9-89ed-4f1a-8dec-602fffd9c072-tmp-dir\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.227302 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.227276 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 23:29:13.227409 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.227327 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 23:29:13.227478 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.227460 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 23:29:13.228376 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.228358 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 23:29:13.229045 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.228892 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k24zf\"" Nov 03 23:29:13.229045 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.228933 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 23:29:13.229045 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.228944 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k4d88\"" Nov 03 23:29:13.235209 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.234592 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h4spw"] Nov 03 23:29:13.236944 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.236921 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqtnl"] Nov 03 23:29:13.259987 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.259963 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6qpvg"] Nov 03 23:29:13.261659 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.261640 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.277891 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.277861 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qpvg"] Nov 03 23:29:13.326262 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.326230 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 23:29:13.326449 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.326283 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 23:29:13.326449 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.326308 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 23:29:13.326989 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.326964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.327108 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327070 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc833b9-89ed-4f1a-8dec-602fffd9c072-tmp-dir\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.327170 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc8829d-b25c-4952-8cd8-816aa35089ac-data-volume\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.327170 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327149 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bj29\" (UniqueName: \"kubernetes.io/projected/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-api-access-9bj29\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.327249 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327190 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 23:29:13.327249 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ht52\" (UniqueName: \"kubernetes.io/projected/17b0bd49-65e6-4393-88ba-b578c5eab313-kube-api-access-9ht52\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:13.327324 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327249 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc833b9-89ed-4f1a-8dec-602fffd9c072-config-volume\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.327324 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327264 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x2bch\"" Nov 03 23:29:13.327324 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327269 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc833b9-89ed-4f1a-8dec-602fffd9c072-metrics-tls\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.327423 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327320 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b0bd49-65e6-4393-88ba-b578c5eab313-cert\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:13.327423 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327376 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt84d\" (UniqueName: \"kubernetes.io/projected/2bc833b9-89ed-4f1a-8dec-602fffd9c072-kube-api-access-pt84d\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.327423 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fc8829d-b25c-4952-8cd8-816aa35089ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.327514 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327448 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fc8829d-b25c-4952-8cd8-816aa35089ac-crio-socket\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.327514 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327464 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc833b9-89ed-4f1a-8dec-602fffd9c072-tmp-dir\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.327867 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.327851 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc833b9-89ed-4f1a-8dec-602fffd9c072-config-volume\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.331273 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.331242 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc833b9-89ed-4f1a-8dec-602fffd9c072-metrics-tls\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.331370 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.331297 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b0bd49-65e6-4393-88ba-b578c5eab313-cert\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:13.339402 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.339377 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt84d\" (UniqueName: \"kubernetes.io/projected/2bc833b9-89ed-4f1a-8dec-602fffd9c072-kube-api-access-pt84d\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.339487 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.339376 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ht52\" (UniqueName: \"kubernetes.io/projected/17b0bd49-65e6-4393-88ba-b578c5eab313-kube-api-access-9ht52\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:13.428619 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.428579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc8829d-b25c-4952-8cd8-816aa35089ac-data-volume\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.428619 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.428622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bj29\" (UniqueName: \"kubernetes.io/projected/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-api-access-9bj29\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.428812 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.428666 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fc8829d-b25c-4952-8cd8-816aa35089ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.428812 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.428684 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fc8829d-b25c-4952-8cd8-816aa35089ac-crio-socket\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.428812 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.428712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.428964 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.428938 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fc8829d-b25c-4952-8cd8-816aa35089ac-crio-socket\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.429018 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.429000 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc8829d-b25c-4952-8cd8-816aa35089ac-data-volume\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.429367 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.429342 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.430892 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.430874 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fc8829d-b25c-4952-8cd8-816aa35089ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.441957 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.441934 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bj29\" (UniqueName: \"kubernetes.io/projected/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-api-access-9bj29\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.532919 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.532837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:13.539303 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.539274 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:13.574977 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.572997 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:13.855967 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.855892 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qpvg"] Nov 03 23:29:13.859309 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.859281 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqtnl"] Nov 03 23:29:13.861769 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:13.861751 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h4spw"] Nov 03 23:29:13.863072 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:13.863045 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b0bd49_65e6_4393_88ba_b578c5eab313.slice/crio-924db680829930627e55d2ab35f80475fc4c9245d0ad2de017a27480b7ed32ba WatchSource:0}: Error finding container 924db680829930627e55d2ab35f80475fc4c9245d0ad2de017a27480b7ed32ba: Status 404 returned error can't find the container with id 924db680829930627e55d2ab35f80475fc4c9245d0ad2de017a27480b7ed32ba Nov 03 23:29:13.864954 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:13.864938 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc833b9_89ed_4f1a_8dec_602fffd9c072.slice/crio-e446a430ade0ce7fe1db989827c1ab118270009eddf69f384faa9a2877ae575c WatchSource:0}: Error finding container e446a430ade0ce7fe1db989827c1ab118270009eddf69f384faa9a2877ae575c: Status 404 returned error can't find the container with id e446a430ade0ce7fe1db989827c1ab118270009eddf69f384faa9a2877ae575c Nov 03 23:29:14.323682 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:14.323648 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vsrwg_1855ee11-b7e8-49f7-bf9d-4439ac060f20/dns-node-resolver/0.log" Nov 03 23:29:14.331920 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:14.331895 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qpvg" event={"ID":"9fc8829d-b25c-4952-8cd8-816aa35089ac","Type":"ContainerStarted","Data":"6b2068ff2025bfd690380f556261c6061ad45e2588160d8e6dd2ab561d8131ce"} Nov 03 23:29:14.332084 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:14.331951 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qpvg" event={"ID":"9fc8829d-b25c-4952-8cd8-816aa35089ac","Type":"ContainerStarted","Data":"60c64add717a314691a3a2a24ef2b8fef80afc6735929b6877642dbd4d2b8755"} Nov 03 23:29:14.333116 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:14.333089 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h4spw" event={"ID":"2bc833b9-89ed-4f1a-8dec-602fffd9c072","Type":"ContainerStarted","Data":"e446a430ade0ce7fe1db989827c1ab118270009eddf69f384faa9a2877ae575c"} Nov 03 23:29:14.334180 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:14.334158 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqtnl" event={"ID":"17b0bd49-65e6-4393-88ba-b578c5eab313","Type":"ContainerStarted","Data":"924db680829930627e55d2ab35f80475fc4c9245d0ad2de017a27480b7ed32ba"} Nov 03 23:29:14.976791 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:14.976745 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:15.100383 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.100295 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:15.100383 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.100355 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:15.100607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.100542 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:15.107184 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.107156 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 23:29:15.107391 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.107232 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 23:29:15.111524 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.111437 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 23:29:15.111524 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.111471 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnnt6\"" Nov 03 23:29:15.111524 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.111484 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnhvl\"" Nov 03 23:29:15.111524 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.111487 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 23:29:15.111723 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.111537 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hkcvs\"" Nov 03 23:29:15.179620 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.179576 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:15.324447 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.324411 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:15.340587 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.340553 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qpvg" event={"ID":"9fc8829d-b25c-4952-8cd8-816aa35089ac","Type":"ContainerStarted","Data":"e4eb662c6149449c3d4e73fe010482d45f1e3866cea59586f2ff7d2aab0ade0d"} Nov 03 23:29:15.431194 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.431120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:15.524926 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:15.524736 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dbjll_58a707ad-9256-4eb1-bf0f-eae7ab6c8d32/node-ca/0.log" Nov 03 23:29:16.315016 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.314975 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:16.335301 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.335268 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl"] Nov 03 23:29:16.337283 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.337244 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.344775 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.344749 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9cdhc\"" Nov 03 23:29:16.347401 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.347376 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 23:29:16.348356 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.348328 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h4spw" event={"ID":"2bc833b9-89ed-4f1a-8dec-602fffd9c072","Type":"ContainerStarted","Data":"b10a4b2449a275fc9b6c5c93de4254df4cc76bbe75a24b550d90a351c48b3478"} Nov 03 23:29:16.348572 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.348390 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h4spw" event={"ID":"2bc833b9-89ed-4f1a-8dec-602fffd9c072","Type":"ContainerStarted","Data":"072ee89a0b1cd8a538ba6b08774abf55b505f5cf459c94e3cde1acc8119ce2d3"} Nov 03 23:29:16.348572 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.348526 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:16.349024 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.348912 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 23:29:16.349024 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.348989 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 23:29:16.349024 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.349011 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 23:29:16.352260 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.352237 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl"] Nov 03 23:29:16.354477 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.354451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.354579 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.354497 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.354643 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.354606 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.354684 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.354639 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp7xk\" (UniqueName: \"kubernetes.io/projected/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-kube-api-access-kp7xk\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.389700 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.389637 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-h4spw" podStartSLOduration=1.439816908 podStartE2EDuration="3.389617068s" totalImagesPullingTime="1.94980016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:16.38916755 +0000 UTC m=+49.078956440" watchObservedRunningTime="2026-11-03 23:29:16.389617068 +0000 UTC m=+49.079405955" Nov 03 23:29:16.421912 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.421884 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 23:29:16.455269 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.455245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.455362 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.455281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.455512 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.455491 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.455608 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.455521 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp7xk\" (UniqueName: \"kubernetes.io/projected/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-kube-api-access-kp7xk\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.455998 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.455971 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.458943 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.458916 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.460594 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.460567 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.470379 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.470354 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp7xk\" (UniqueName: \"kubernetes.io/projected/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-kube-api-access-kp7xk\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.472515 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.472493 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:16.573329 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.573250 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:16.655923 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.655888 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:16.939640 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:16.939564 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl"] Nov 03 23:29:16.943005 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:16.942981 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f77e6f_4bd8_4a9e_ab23_c1337365cac1.slice/crio-f1b9403b98d9df347f27b95a3c4ecab5e1938f10d7e18b5af052c7ec9f80a4c5 WatchSource:0}: Error finding container f1b9403b98d9df347f27b95a3c4ecab5e1938f10d7e18b5af052c7ec9f80a4c5: Status 404 returned error can't find the container with id f1b9403b98d9df347f27b95a3c4ecab5e1938f10d7e18b5af052c7ec9f80a4c5 Nov 03 23:29:17.279302 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.279264 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b687d5469-mzgl6"] Nov 03 23:29:17.282316 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.282298 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.291810 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.291778 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 23:29:17.292251 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.292234 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 23:29:17.292501 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.292488 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 23:29:17.292501 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.292496 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 23:29:17.292603 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.292535 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 23:29:17.292647 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.292599 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 23:29:17.293156 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.293142 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hpfml\"" Nov 03 23:29:17.293206 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.293156 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 23:29:17.299811 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.299787 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 23:29:17.300495 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.300471 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b687d5469-mzgl6"] Nov 03 23:29:17.352776 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.352743 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qpvg" event={"ID":"9fc8829d-b25c-4952-8cd8-816aa35089ac","Type":"ContainerStarted","Data":"c2932d68829aa16ea209d91babf8a496e59b3cac026706efc9b2d5ba27c545e4"} Nov 03 23:29:17.354027 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.354001 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" event={"ID":"37f77e6f-4bd8-4a9e-ab23-c1337365cac1","Type":"ContainerStarted","Data":"f1b9403b98d9df347f27b95a3c4ecab5e1938f10d7e18b5af052c7ec9f80a4c5"} Nov 03 23:29:17.355508 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.355487 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqtnl" event={"ID":"17b0bd49-65e6-4393-88ba-b578c5eab313","Type":"ContainerStarted","Data":"6b72a96fc69c4917563f7de668f157e8e563fe8383f7245ec22b1813c5d93a95"} Nov 03 23:29:17.362563 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.362537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.362697 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.362572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.362697 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.362611 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cz7\" (UniqueName: \"kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.362779 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.362708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.362779 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.362751 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.362851 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.362800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.362851 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.362821 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.362851 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.362838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.376422 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.376371 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6qpvg" podStartSLOduration=1.914428672 podStartE2EDuration="4.376356927s" totalImagesPullingTime="2.461928255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:29:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:17.374810921 +0000 UTC m=+50.064599801" watchObservedRunningTime="2026-11-03 23:29:17.376356927 +0000 UTC m=+50.066145807" Nov 03 23:29:17.401962 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.401899 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pqtnl" podStartSLOduration=1.385592617 podStartE2EDuration="4.401886783s" totalImagesPullingTime="3.016294166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:17.401165593 +0000 UTC m=+50.090954494" watchObservedRunningTime="2026-11-03 23:29:17.401886783 +0000 UTC m=+50.091675668" Nov 03 23:29:17.464086 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.464028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cz7\" (UniqueName: \"kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.464086 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.464082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.464302 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.464108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.464302 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.464279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.464383 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.464369 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.464439 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.464394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.464536 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.464515 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.464634 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.464553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.465295 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.465235 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.465295 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.465255 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.465295 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.465287 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.465712 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.465683 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.466512 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.466489 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.466938 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.466917 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.467046 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.467018 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.484216 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.484192 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cz7\" (UniqueName: \"kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.593844 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.593759 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:17.887235 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:17.887148 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b687d5469-mzgl6"] Nov 03 23:29:17.890939 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:17.890908 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bfaf2a_e644_40b1_84cd_f2d7d0417207.slice/crio-bce424346a312c6e62eaa196c1114b1df85e40966486216b215ab9b8b02034af WatchSource:0}: Error finding container bce424346a312c6e62eaa196c1114b1df85e40966486216b215ab9b8b02034af: Status 404 returned error can't find the container with id bce424346a312c6e62eaa196c1114b1df85e40966486216b215ab9b8b02034af Nov 03 23:29:18.360456 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:18.360427 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" event={"ID":"37f77e6f-4bd8-4a9e-ab23-c1337365cac1","Type":"ContainerStarted","Data":"f6841c55de75bf7f27e82b250e3ab37fb94fbce62601e25b4a56af175b47746d"} Nov 03 23:29:18.361738 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:18.361715 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b687d5469-mzgl6" event={"ID":"d8bfaf2a-e644-40b1-84cd-f2d7d0417207","Type":"ContainerStarted","Data":"bce424346a312c6e62eaa196c1114b1df85e40966486216b215ab9b8b02034af"} Nov 03 23:29:18.368985 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:18.368957 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:18.599149 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:18.599013 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:18.740537 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:18.740488 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:18.964306 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:18.964268 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:19.372209 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:19.372019 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" event={"ID":"37f77e6f-4bd8-4a9e-ab23-c1337365cac1","Type":"ContainerStarted","Data":"effafe3b42adee5de9bf5b7dc95d69d8ff0088d6f4f160abd94a8d73dedf77c7"} Nov 03 23:29:19.393551 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:19.393490 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" podStartSLOduration=2.054985286 podStartE2EDuration="3.393471528s" totalImagesPullingTime="1.338486242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:19.393105513 +0000 UTC m=+52.082894396" watchObservedRunningTime="2026-11-03 23:29:19.393471528 +0000 UTC m=+52.083260417" Nov 03 23:29:21.012256 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.012220 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:21.113135 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.113102 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:21.380501 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.380416 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b687d5469-mzgl6" event={"ID":"d8bfaf2a-e644-40b1-84cd-f2d7d0417207","Type":"ContainerStarted","Data":"928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f"} Nov 03 23:29:21.434005 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.433952 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b687d5469-mzgl6" podStartSLOduration=1.65584263 podStartE2EDuration="4.433938738s" totalImagesPullingTime="2.778096108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:21.432640218 +0000 UTC m=+54.122429114" watchObservedRunningTime="2026-11-03 23:29:21.433938738 +0000 UTC m=+54.123727624" Nov 03 23:29:21.584965 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.584922 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:21.702175 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.702088 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:21.861907 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.861871 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:21.943140 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.943108 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dsslm"] Nov 03 23:29:21.946271 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.946254 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:21.954608 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.954564 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fw76h\"" Nov 03 23:29:21.957739 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.957721 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 23:29:21.957805 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.957762 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 23:29:21.990417 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.990392 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 23:29:21.999128 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:21.999089 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:22.000231 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.000195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.000231 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.000236 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wfwh\" (UniqueName: \"kubernetes.io/projected/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-kube-api-access-8wfwh\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.000375 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.000256 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.000412 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.000390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-metrics-client-ca\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.000449 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.000418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-sys\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.000490 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.000448 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-root\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.000541 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.000522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-wtmp\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.000581 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.000569 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-tls\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.000647 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.000632 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-textfile\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101109 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-metrics-client-ca\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101109 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-sys\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101147 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-root\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101208 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-root\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101218 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-sys\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101261 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-wtmp\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-tls\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101362 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-textfile\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101426 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wfwh\" (UniqueName: \"kubernetes.io/projected/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-kube-api-access-8wfwh\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101441 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-wtmp\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.101607 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101458 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.102006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101735 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-metrics-client-ca\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.102006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101800 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-textfile\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.102006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.101989 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.103785 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.103743 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.103908 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.103835 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-tls\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.113961 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.113942 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wfwh\" (UniqueName: \"kubernetes.io/projected/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-kube-api-access-8wfwh\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.141433 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.141410 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:22.260799 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.260764 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:22.261887 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.261858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:22.271494 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:22.271472 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc30eef_1cd6_449c_9aef_d15732f1a2f8.slice/crio-42b2b5c55361355087f672ab789b67964e6751c7865bb7f83508e3e5c6b44913 WatchSource:0}: Error finding container 42b2b5c55361355087f672ab789b67964e6751c7865bb7f83508e3e5c6b44913: Status 404 returned error can't find the container with id 42b2b5c55361355087f672ab789b67964e6751c7865bb7f83508e3e5c6b44913 Nov 03 23:29:22.389649 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.389615 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsslm" event={"ID":"fdc30eef-1cd6-449c-9aef-d15732f1a2f8","Type":"ContainerStarted","Data":"42b2b5c55361355087f672ab789b67964e6751c7865bb7f83508e3e5c6b44913"} Nov 03 23:29:22.460476 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.460439 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:22.566834 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.566753 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:22.670181 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.670155 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:22.769732 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.769701 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:22.905826 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:22.905740 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:23.082162 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:23.082125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:24.300006 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:24.299979 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:24.396968 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:24.396933 2570 generic.go:356] "Generic (PLEG): container finished" podID="fdc30eef-1cd6-449c-9aef-d15732f1a2f8" containerID="13ae612c564211ab1e87732dfcb4ab392ef6a5a46a3f8533a560d93c03784d17" exitCode=0 Nov 03 23:29:24.397142 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:24.397008 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsslm" event={"ID":"fdc30eef-1cd6-449c-9aef-d15732f1a2f8","Type":"ContainerDied","Data":"13ae612c564211ab1e87732dfcb4ab392ef6a5a46a3f8533a560d93c03784d17"} Nov 03 23:29:24.472145 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:24.472116 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:24.563805 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:24.563716 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:24.786564 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:24.786525 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:24.895380 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:24.895293 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:25.014106 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:25.014068 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:25.111657 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:25.111592 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:25.402697 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:25.402662 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsslm" event={"ID":"fdc30eef-1cd6-449c-9aef-d15732f1a2f8","Type":"ContainerStarted","Data":"03474d5a417411b50a18555acb77aeb78a076954ba4d2143a22c2235ab6962f3"} Nov 03 23:29:25.403122 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:25.402712 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dsslm" event={"ID":"fdc30eef-1cd6-449c-9aef-d15732f1a2f8","Type":"ContainerStarted","Data":"8ee592291f7ecd3612313a6cc6c73a8e490a7269142d36bc7bd1caf3aa269969"} Nov 03 23:29:25.427810 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:25.427753 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dsslm" podStartSLOduration=3.290538272 podStartE2EDuration="4.427735644s" totalImagesPullingTime="1.073268851s" totalInitContainerRuntime="63.928521ms" isStatefulPod=true podCreationTimestamp="2026-11-03 23:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:25.42545685 +0000 UTC m=+58.115245736" watchObservedRunningTime="2026-11-03 23:29:25.427735644 +0000 UTC m=+58.117524529" Nov 03 23:29:26.358251 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:26.358220 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:26.598248 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:26.598212 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:26.699778 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:26.699695 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:27.130938 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.130901 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5df744b67-frfj2"] Nov 03 23:29:27.134042 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.134020 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.142690 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.142667 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 23:29:27.142820 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.142773 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 23:29:27.142820 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.142817 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 23:29:27.142918 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.142877 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 23:29:27.142969 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.142929 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 23:29:27.142969 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.142935 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z6dff\"" Nov 03 23:29:27.148016 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.147998 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 23:29:27.157328 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.157306 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5df744b67-frfj2"] Nov 03 23:29:27.242852 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.242812 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.242852 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.242852 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.243098 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.242875 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-federate-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.243098 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.242908 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.243098 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.243021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2zm\" (UniqueName: \"kubernetes.io/projected/3a95c441-120e-44ce-8b68-516030f94a91-kube-api-access-9h2zm\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.243098 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.243073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.243221 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.243150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-serving-certs-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.243221 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.243198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-metrics-client-ca\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.312663 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.312630 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:27.344069 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.343997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-serving-certs-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.344069 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.344062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-metrics-client-ca\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.344286 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.344085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.344286 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.344105 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.344286 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.344202 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-federate-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.344286 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.344257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.344488 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.344325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2zm\" (UniqueName: \"kubernetes.io/projected/3a95c441-120e-44ce-8b68-516030f94a91-kube-api-access-9h2zm\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.344488 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.344459 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.344886 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.344859 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-serving-certs-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.345175 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.345152 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-metrics-client-ca\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.345275 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.345197 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.346824 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.346793 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.346824 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.346805 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-federate-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.347020 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.347001 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.347087 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.347004 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.357065 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.357026 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2zm\" (UniqueName: \"kubernetes.io/projected/3a95c441-120e-44ce-8b68-516030f94a91-kube-api-access-9h2zm\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.406875 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.406788 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:27.448614 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.448579 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:27.594670 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.594631 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:27.594851 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.594704 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:27.599329 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.599300 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:27.704904 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.704822 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:27.728395 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.728358 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5df744b67-frfj2"] Nov 03 23:29:27.732633 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:27.732604 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a95c441_120e_44ce_8b68_516030f94a91.slice/crio-757121206c662fd3568284fc55097fefc9bb916b12fafdaf41b494e17e64351f WatchSource:0}: Error finding container 757121206c662fd3568284fc55097fefc9bb916b12fafdaf41b494e17e64351f: Status 404 returned error can't find the container with id 757121206c662fd3568284fc55097fefc9bb916b12fafdaf41b494e17e64351f Nov 03 23:29:27.812446 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:27.812411 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:28.009560 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.009522 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:28.155394 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.155359 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 23:29:28.160184 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.160155 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.168009 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.167987 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 23:29:28.174254 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174231 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 23:29:28.174450 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174299 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 23:29:28.174450 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174299 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 23:29:28.174450 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174406 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cfq6h\"" Nov 03 23:29:28.174450 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174422 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 23:29:28.174450 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174432 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 23:29:28.174450 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174435 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 23:29:28.174802 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174477 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 23:29:28.174802 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174497 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-482vvasamrjf7\"" Nov 03 23:29:28.174802 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174537 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 23:29:28.174802 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.174574 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 23:29:28.185133 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.185101 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 23:29:28.197220 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.197189 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:28.253692 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.253646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.253875 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.253714 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.253875 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.253799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.253875 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.253836 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.253875 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.253853 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254107 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.253899 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254107 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254019 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254107 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254248 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254114 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254248 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254139 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnbk\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254248 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254360 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254270 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254360 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254288 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254360 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254306 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254360 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254478 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254382 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254478 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254408 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254478 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254456 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254562 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.254591 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.254540 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.270408 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.270334 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 23:29:28.270408 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.270384 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 23:29:28.270558 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.270430 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 23:29:28.270558 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.270456 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 23:29:28.274400 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.274376 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 23:29:28.320922 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.320888 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:28.355303 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355267 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.355461 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.355461 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.355461 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355366 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.355461 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.355461 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355435 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.355461 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355453 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.355863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.355863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355515 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.355987 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.355962 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356069 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356130 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356179 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356126 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356179 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356154 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356276 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356276 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356377 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356377 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356302 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnbk\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356377 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356316 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356377 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356341 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356563 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356380 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356563 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356411 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356563 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356554 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.356960 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.356935 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.357160 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.357096 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.357654 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.357626 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.358469 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.358443 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.359218 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.359170 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.361218 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.359701 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.361218 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.359720 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.361218 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.360228 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.361218 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.360370 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.361218 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.360600 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.361218 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.361180 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.361218 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.361212 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.362243 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.362220 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.362468 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.362447 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.362792 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.362772 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.364066 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.364049 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.372343 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.372290 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnbk\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.417150 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.417107 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" event={"ID":"3a95c441-120e-44ce-8b68-516030f94a91","Type":"ContainerStarted","Data":"757121206c662fd3568284fc55097fefc9bb916b12fafdaf41b494e17e64351f"} Nov 03 23:29:28.422182 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.422152 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:28.490074 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.489970 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:28.793720 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:28.793681 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 23:29:28.797738 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:28.797707 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf866b162_3fe8_4a69_9ae0_387e0c18d16d.slice/crio-b5f3d49f7567843143cc40a2ee82a15d7349002d5fca297f2e2bd6c81a1d6eca WatchSource:0}: Error finding container b5f3d49f7567843143cc40a2ee82a15d7349002d5fca297f2e2bd6c81a1d6eca: Status 404 returned error can't find the container with id b5f3d49f7567843143cc40a2ee82a15d7349002d5fca297f2e2bd6c81a1d6eca Nov 03 23:29:29.421889 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.421848 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerStarted","Data":"b5f3d49f7567843143cc40a2ee82a15d7349002d5fca297f2e2bd6c81a1d6eca"} Nov 03 23:29:29.487002 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.486973 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c96f49478-btlbq"] Nov 03 23:29:29.489946 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.489923 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.511337 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.511303 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c96f49478-btlbq"] Nov 03 23:29:29.582753 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.582711 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.582948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.582761 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.582948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.582790 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.582948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.582843 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.582948 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.582906 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.583173 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.582971 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.583173 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.583009 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.583173 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.583048 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfzcx\" (UniqueName: \"kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.684197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.684103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.684197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.684155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzcx\" (UniqueName: \"kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.684197 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.684187 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.684477 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.684217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.684477 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.684241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.684477 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.684294 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.684477 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.684342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.684477 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.684393 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.686290 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.685550 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.686290 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.685658 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.686290 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.685769 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.686290 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.686133 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.686290 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.686202 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.687433 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.687396 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.687544 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.687522 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.699210 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.699177 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzcx\" (UniqueName: \"kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.801576 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.801533 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:29.918169 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:29.918137 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:30.016170 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.016131 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:30.088009 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.087977 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c96f49478-btlbq"] Nov 03 23:29:30.096096 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:30.096072 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38365ee_cbdb_4999_a6c9_e78aa4552bb9.slice/crio-92c0ec3dbccb50c66767557717dd84cdb65d12e65edd70a84cd706b09493716a WatchSource:0}: Error finding container 92c0ec3dbccb50c66767557717dd84cdb65d12e65edd70a84cd706b09493716a: Status 404 returned error can't find the container with id 92c0ec3dbccb50c66767557717dd84cdb65d12e65edd70a84cd706b09493716a Nov 03 23:29:30.359467 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.359417 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:30.427418 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.427382 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" event={"ID":"3a95c441-120e-44ce-8b68-516030f94a91","Type":"ContainerStarted","Data":"4b1c6ae74e20f4fa6c796789d6f98a0faabb585106092fe16aaeb8721b90eca4"} Nov 03 23:29:30.427621 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.427445 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" event={"ID":"3a95c441-120e-44ce-8b68-516030f94a91","Type":"ContainerStarted","Data":"db1ef34d04ee834916ad5cdf40e0ce54f97480513718928e1879eb5b8227773d"} Nov 03 23:29:30.427621 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.427471 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" event={"ID":"3a95c441-120e-44ce-8b68-516030f94a91","Type":"ContainerStarted","Data":"802e14be0fd754977d92103711fda779e1fc9e38738dd4572edc08e1511a30a6"} Nov 03 23:29:30.429001 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.428970 2570 generic.go:356] "Generic (PLEG): container finished" podID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerID="7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f" exitCode=0 Nov 03 23:29:30.429103 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.429029 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerDied","Data":"7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f"} Nov 03 23:29:30.430617 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.430593 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c96f49478-btlbq" event={"ID":"c38365ee-cbdb-4999-a6c9-e78aa4552bb9","Type":"ContainerStarted","Data":"c2d5eadc419d726416fde7d70cfc7cb9cd8f80edc150149569fe098c5ef95443"} Nov 03 23:29:30.430695 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.430634 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c96f49478-btlbq" event={"ID":"c38365ee-cbdb-4999-a6c9-e78aa4552bb9","Type":"ContainerStarted","Data":"92c0ec3dbccb50c66767557717dd84cdb65d12e65edd70a84cd706b09493716a"} Nov 03 23:29:30.454749 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.454700 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" podStartSLOduration=1.082826234 podStartE2EDuration="3.454685061s" totalImagesPullingTime="2.371858827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:30.452025455 +0000 UTC m=+63.141814350" watchObservedRunningTime="2026-11-03 23:29:30.454685061 +0000 UTC m=+63.144473991" Nov 03 23:29:30.466215 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.466178 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:30.509080 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:30.509006 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c96f49478-btlbq" podStartSLOduration=1.5089929149999999 podStartE2EDuration="1.508992915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:30.508434005 +0000 UTC m=+63.198222915" watchObservedRunningTime="2026-11-03 23:29:30.508992915 +0000 UTC m=+63.198781802" Nov 03 23:29:31.353624 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:31.353588 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:31.480797 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:31.480763 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:32.355293 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:32.355259 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:32.453807 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:32.453776 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:33.523412 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.523379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:33.523843 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.523521 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:33.526772 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.526744 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 23:29:33.528055 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.528015 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 23:29:33.536940 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.536910 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:33.537212 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.537191 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:33.717861 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.717830 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnnt6\"" Nov 03 23:29:33.724585 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.724551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:33.724776 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.724651 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:33.727131 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.727102 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnhvl\"" Nov 03 23:29:33.727622 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.727603 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 23:29:33.734010 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.733983 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:33.738503 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.738201 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 23:29:33.749025 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:33.748999 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:34.023028 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.022989 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p7zhg"] Nov 03 23:29:34.027965 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:34.027940 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c784845_c509_4b51_86b8_ce9075600208.slice/crio-ef745d51de9b1e8bdaf3a77891a4bc05097a77ef09c18527b8d5963f28649b4b WatchSource:0}: Error finding container ef745d51de9b1e8bdaf3a77891a4bc05097a77ef09c18527b8d5963f28649b4b: Status 404 returned error can't find the container with id ef745d51de9b1e8bdaf3a77891a4bc05097a77ef09c18527b8d5963f28649b4b Nov 03 23:29:34.032938 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.032917 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hkcvs\"" Nov 03 23:29:34.038417 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.038392 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-chg92"] Nov 03 23:29:34.041003 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.040739 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:34.082716 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.082681 2570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c96f49478-btlbq"] Nov 03 23:29:34.118960 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.118929 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-854967fbd5-4lmhp"] Nov 03 23:29:34.123207 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.123182 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.134594 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.134567 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854967fbd5-4lmhp"] Nov 03 23:29:34.232185 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.232155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.232311 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.232191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.232311 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.232215 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.232311 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.232234 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.232311 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.232294 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.232515 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.232461 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.232570 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.232527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44cj\" (UniqueName: \"kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.232570 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.232562 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.333069 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.332980 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s7hm2"] Nov 03 23:29:34.333346 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.333329 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.333389 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.333358 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44cj\" (UniqueName: \"kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.333389 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.333378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.333498 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.333421 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.333498 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.333448 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.333498 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.333476 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.333628 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.333503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.333790 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.333762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.333892 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.333870 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.335718 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.335691 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.335863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.335816 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.335863 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.335837 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.335987 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.335968 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.337358 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:34.337328 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d93d4f_2af9_44d5_b8e5_542b30760f34.slice/crio-b9cb916fd007a09e818c1897a867326f80e0ea794ccb9d87872b92702fa2f8f6 WatchSource:0}: Error finding container b9cb916fd007a09e818c1897a867326f80e0ea794ccb9d87872b92702fa2f8f6: Status 404 returned error can't find the container with id b9cb916fd007a09e818c1897a867326f80e0ea794ccb9d87872b92702fa2f8f6 Nov 03 23:29:34.337950 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.337926 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.338018 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.337931 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.363545 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.363516 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44cj\" (UniqueName: \"kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.436334 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.436301 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:34.446093 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.446058 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p7zhg" event={"ID":"9c784845-c509-4b51-86b8-ce9075600208","Type":"ContainerStarted","Data":"ef745d51de9b1e8bdaf3a77891a4bc05097a77ef09c18527b8d5963f28649b4b"} Nov 03 23:29:34.448323 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.448293 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerStarted","Data":"2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457"} Nov 03 23:29:34.448445 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.448361 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerStarted","Data":"9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5"} Nov 03 23:29:34.449418 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.449395 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s7hm2" event={"ID":"72d93d4f-2af9-44d5-b8e5-542b30760f34","Type":"ContainerStarted","Data":"b9cb916fd007a09e818c1897a867326f80e0ea794ccb9d87872b92702fa2f8f6"} Nov 03 23:29:34.450405 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.450382 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-chg92" event={"ID":"f42ad3cc-de80-43bb-b548-dfa6c78618e5","Type":"ContainerStarted","Data":"e1e96e41f0a73939ebffa16f757381cf9c8b4ab31bad667c62795b5f36d4b1db"} Nov 03 23:29:34.471646 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.471619 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:34.592739 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.592655 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:34.712892 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.712858 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854967fbd5-4lmhp"] Nov 03 23:29:34.715405 ip-10-0-138-137 kubenswrapper[2570]: W1103 23:29:34.715377 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53faae9_d5e5_4c53_8f4c_1f421e40f203.slice/crio-38ce56ebdf0ca36e6f651f549321a729bcd596b712616053e74b8ca532d613c3 WatchSource:0}: Error finding container 38ce56ebdf0ca36e6f651f549321a729bcd596b712616053e74b8ca532d613c3: Status 404 returned error can't find the container with id 38ce56ebdf0ca36e6f651f549321a729bcd596b712616053e74b8ca532d613c3 Nov 03 23:29:34.928588 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:34.928514 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:35.031781 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:35.031749 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:35.137244 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:35.137200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:35.468859 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:35.468819 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854967fbd5-4lmhp" event={"ID":"f53faae9-d5e5-4c53-8f4c-1f421e40f203","Type":"ContainerStarted","Data":"ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439"} Nov 03 23:29:35.469015 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:35.468879 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854967fbd5-4lmhp" event={"ID":"f53faae9-d5e5-4c53-8f4c-1f421e40f203","Type":"ContainerStarted","Data":"38ce56ebdf0ca36e6f651f549321a729bcd596b712616053e74b8ca532d613c3"} Nov 03 23:29:35.472494 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:35.472463 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-chg92" event={"ID":"f42ad3cc-de80-43bb-b548-dfa6c78618e5","Type":"ContainerStarted","Data":"f52d5685e57600d7d19549f4edc555ca1bc5c0aa477abeb1e4d3f5c8d95be3a8"} Nov 03 23:29:35.514913 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:35.514871 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-854967fbd5-4lmhp" podStartSLOduration=1.514855675 podStartE2EDuration="1.514855675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:35.513539975 +0000 UTC m=+68.203328860" watchObservedRunningTime="2026-11-03 23:29:35.514855675 +0000 UTC m=+68.204644561" Nov 03 23:29:35.573476 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:35.573229 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:35.675577 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:35.675537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:36.480773 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:36.480731 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-chg92" event={"ID":"f42ad3cc-de80-43bb-b548-dfa6c78618e5","Type":"ContainerStarted","Data":"a641d976100bdbbdc3772e150d633aae4dcac787a032b070a7d78ac76e71d556"} Nov 03 23:29:36.504061 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:36.502697 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-chg92" podStartSLOduration=67.235529453 podStartE2EDuration="1m8.502679749s" totalImagesPullingTime="1.267150296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:36.502479678 +0000 UTC m=+69.192268576" watchObservedRunningTime="2026-11-03 23:29:36.502679749 +0000 UTC m=+69.192468634" Nov 03 23:29:36.633310 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:36.633274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:36.758430 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:36.758237 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:36.864750 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:36.864700 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:37.486951 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:37.486923 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerStarted","Data":"cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58"} Nov 03 23:29:37.487140 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:37.486993 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerStarted","Data":"966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20"} Nov 03 23:29:37.487140 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:37.487069 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerStarted","Data":"5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e"} Nov 03 23:29:38.492880 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:38.492835 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerStarted","Data":"d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a"} Nov 03 23:29:38.494429 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:38.494395 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s7hm2" event={"ID":"72d93d4f-2af9-44d5-b8e5-542b30760f34","Type":"ContainerStarted","Data":"685afd977816acfaaa40d5cf8f48cd41d67154fd7cdcefc4b52be7ef73efd906"} Nov 03 23:29:38.494528 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:38.494513 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:38.535977 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:38.535910 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5391762 podStartE2EDuration="10.535887956s" totalImagesPullingTime="7.974278271s" totalInitContainerRuntime="22.433485ms" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:38.532835728 +0000 UTC m=+71.222624657" watchObservedRunningTime="2026-11-03 23:29:38.535887956 +0000 UTC m=+71.225676843" Nov 03 23:29:38.552454 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:38.552401 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s7hm2" podStartSLOduration=66.690990074 podStartE2EDuration="1m10.552385965s" totalImagesPullingTime="3.861395891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:38.551152763 +0000 UTC m=+71.240941661" watchObservedRunningTime="2026-11-03 23:29:38.552385965 +0000 UTC m=+71.242174851" Nov 03 23:29:39.614807 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:39.614772 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:39.783685 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:39.783651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:39.802235 ip-10-0-138-137 kubenswrapper[2570]: I1103 23:29:39.802209 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:41.101770 ip-10-0-138-137 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 23:29:41.109837 ip-10-0-138-137 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 23:29:41.110000 ip-10-0-138-137 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 23:29:41.110073 ip-10-0-138-137 systemd[1]: kubelet.service: Consumed 4.363s CPU time. Nov 03 23:29:41.127577 ip-10-0-138-137 systemd[1]: Starting Kubernetes Kubelet... Nov 03 23:29:41.209933 ip-10-0-138-137 kubenswrapper[6208]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 23:29:41.209933 ip-10-0-138-137 kubenswrapper[6208]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 23:29:41.209933 ip-10-0-138-137 kubenswrapper[6208]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 23:29:41.209933 ip-10-0-138-137 kubenswrapper[6208]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 23:29:41.212060 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.211961 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:29:41.212060 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212059 6208 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:29:41.212060 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212063 6208 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212068 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212071 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212073 6208 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212076 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212079 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212081 6208 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212084 6208 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212087 6208 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212089 6208 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212092 6208 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212094 6208 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212097 6208 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212100 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212102 6208 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212105 6208 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212107 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212110 6208 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212112 6208 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212115 6208 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:29:41.212162 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212118 6208 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212120 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212123 6208 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212129 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212132 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212135 6208 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212138 6208 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212141 6208 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212143 6208 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212147 6208 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212149 6208 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212152 6208 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212154 6208 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212157 6208 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212160 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212162 6208 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212165 6208 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212168 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212171 6208 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212173 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:29:41.212655 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212176 6208 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212179 6208 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212182 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212184 6208 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212187 6208 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212190 6208 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212192 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212195 6208 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212200 6208 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212204 6208 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212207 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212210 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212213 6208 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212216 6208 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212219 6208 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212222 6208 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212224 6208 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212227 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212230 6208 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:29:41.213197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212233 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212235 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212239 6208 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212242 6208 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212244 6208 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212249 6208 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212253 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212256 6208 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212259 6208 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212262 6208 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212265 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212267 6208 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212270 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212273 6208 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212275 6208 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212278 6208 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212281 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212283 6208 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212286 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212289 6208 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:29:41.213652 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212292 6208 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212294 6208 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212297 6208 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212299 6208 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212302 6208 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212305 6208 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212308 6208 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212310 6208 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212313 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212316 6208 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212318 6208 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212321 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212324 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212327 6208 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212330 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212332 6208 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212335 6208 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212337 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212340 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:29:41.214168 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212342 6208 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212345 6208 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212347 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212350 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212353 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.212355 6208 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213136 6208 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213144 6208 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213147 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213150 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213153 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213157 6208 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213159 6208 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213162 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213165 6208 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213167 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213170 6208 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213172 6208 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213175 6208 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:29:41.214724 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213177 6208 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213180 6208 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213183 6208 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213186 6208 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213188 6208 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213193 6208 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213195 6208 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213198 6208 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213201 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213204 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213207 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213210 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213212 6208 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213215 6208 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213218 6208 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213220 6208 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213223 6208 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213225 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213228 6208 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213231 6208 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:29:41.215197 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213234 6208 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213236 6208 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213239 6208 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213242 6208 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213245 6208 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213247 6208 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213250 6208 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213252 6208 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213255 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213258 6208 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213260 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213263 6208 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213265 6208 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213267 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213271 6208 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213273 6208 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213276 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213279 6208 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213281 6208 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213284 6208 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:29:41.215696 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213287 6208 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213289 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213292 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213294 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213297 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213300 6208 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213302 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213305 6208 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213307 6208 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213309 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213312 6208 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213315 6208 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213318 6208 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213320 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213323 6208 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213325 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213328 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213330 6208 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213333 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:29:41.216208 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213336 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213338 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213341 6208 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213343 6208 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213346 6208 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213348 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213351 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213356 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213359 6208 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213362 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213365 6208 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213370 6208 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213374 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213378 6208 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213381 6208 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213385 6208 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213387 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213390 6208 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213393 6208 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:29:41.216703 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213395 6208 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213398 6208 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213400 6208 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213403 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213405 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213408 6208 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213411 6208 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213414 6208 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213416 6208 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213419 6208 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213422 6208 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213427 6208 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213431 6208 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213434 6208 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213437 6208 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213950 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213956 6208 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213960 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213964 6208 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213967 6208 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:29:41.217187 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213972 6208 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213975 6208 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213977 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213980 6208 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213983 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213987 6208 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213989 6208 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213992 6208 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213994 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.213997 6208 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214000 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214002 6208 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214005 6208 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214008 6208 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214011 6208 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214013 6208 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214016 6208 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214018 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214021 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214024 6208 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:29:41.217663 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214027 6208 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214042 6208 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214045 6208 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214048 6208 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214050 6208 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214053 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214056 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214059 6208 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214061 6208 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214064 6208 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214066 6208 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214069 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214074 6208 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214076 6208 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214079 6208 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214089 6208 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214093 6208 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214097 6208 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214100 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214102 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:29:41.218161 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214105 6208 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214108 6208 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214110 6208 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214113 6208 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214116 6208 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214118 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214121 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214124 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214127 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214129 6208 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214132 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214134 6208 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214138 6208 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214141 6208 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214143 6208 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214146 6208 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214148 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214151 6208 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214154 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:29:41.218646 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214157 6208 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214159 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214162 6208 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214164 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214167 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214172 6208 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214174 6208 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214177 6208 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214180 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214183 6208 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214186 6208 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214188 6208 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214191 6208 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214194 6208 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214196 6208 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214199 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214201 6208 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214204 6208 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214207 6208 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:29:41.219124 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214209 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214212 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214214 6208 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214217 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214219 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214222 6208 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214224 6208 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214227 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214230 6208 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214232 6208 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214235 6208 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214238 6208 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214240 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214243 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214247 6208 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214251 6208 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214254 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214257 6208 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214260 6208 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:29:41.219595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214263 6208 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214265 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214268 6208 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.214271 6208 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214371 6208 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214387 6208 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214397 6208 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214402 6208 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214407 6208 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214410 6208 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214415 6208 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214419 6208 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214422 6208 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214425 6208 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214429 6208 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214432 6208 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214435 6208 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214438 6208 flags.go:64] FLAG: --cgroup-root="" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214441 6208 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214445 6208 flags.go:64] FLAG: --client-ca-file="" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214447 6208 flags.go:64] FLAG: --cloud-provider="external" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214451 6208 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214455 6208 flags.go:64] FLAG: --cluster-domain="" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214459 6208 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 23:29:41.220097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214462 6208 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214465 6208 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214469 6208 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214474 6208 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214477 6208 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214480 6208 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214484 6208 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214487 6208 flags.go:64] FLAG: --contention-profiling="false" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214490 6208 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214493 6208 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214497 6208 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214500 6208 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214504 6208 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214507 6208 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214510 6208 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214513 6208 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214516 6208 flags.go:64] FLAG: --enable-server="true" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214519 6208 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214524 6208 flags.go:64] FLAG: --event-burst="100" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214527 6208 flags.go:64] FLAG: --event-qps="50" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214531 6208 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214534 6208 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214537 6208 flags.go:64] FLAG: --eviction-hard="" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214542 6208 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 23:29:41.220666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214545 6208 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214549 6208 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214552 6208 flags.go:64] FLAG: --eviction-soft="" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214555 6208 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214558 6208 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214561 6208 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214565 6208 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214568 6208 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214571 6208 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214574 6208 flags.go:64] FLAG: --feature-gates="" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214578 6208 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214582 6208 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214585 6208 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214588 6208 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214591 6208 flags.go:64] FLAG: --healthz-port="10248" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214594 6208 flags.go:64] FLAG: --help="false" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214597 6208 flags.go:64] FLAG: --hostname-override="ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214601 6208 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214604 6208 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214607 6208 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214610 6208 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214614 6208 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214617 6208 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 23:29:41.221273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214620 6208 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214622 6208 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214625 6208 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214628 6208 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214631 6208 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214634 6208 flags.go:64] FLAG: --kube-reserved="" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214637 6208 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214640 6208 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214643 6208 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214646 6208 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214649 6208 flags.go:64] FLAG: --lock-file="" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214652 6208 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214655 6208 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214658 6208 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214664 6208 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214667 6208 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214670 6208 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214673 6208 flags.go:64] FLAG: --logging-format="text" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214676 6208 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214682 6208 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214685 6208 flags.go:64] FLAG: --manifest-url="" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214687 6208 flags.go:64] FLAG: --manifest-url-header="" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214692 6208 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214695 6208 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214699 6208 flags.go:64] FLAG: --max-pods="110" Nov 03 23:29:41.221845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214702 6208 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214706 6208 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214709 6208 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214712 6208 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214715 6208 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214717 6208 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214720 6208 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214728 6208 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214731 6208 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214734 6208 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214737 6208 flags.go:64] FLAG: --pod-cidr="" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214741 6208 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214744 6208 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214747 6208 flags.go:64] FLAG: --pods-per-core="0" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214750 6208 flags.go:64] FLAG: --port="10250" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214753 6208 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214756 6208 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ca3b396a1af29bd1" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214759 6208 flags.go:64] FLAG: --qos-reserved="" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214762 6208 flags.go:64] FLAG: --read-only-port="10255" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214765 6208 flags.go:64] FLAG: --register-node="true" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214769 6208 flags.go:64] FLAG: --register-with-taints="" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214773 6208 flags.go:64] FLAG: --registry-burst="10" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214776 6208 flags.go:64] FLAG: --registry-qps="5" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214779 6208 flags.go:64] FLAG: --reserved-cpus="" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214782 6208 flags.go:64] FLAG: --reserved-memory="" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214786 6208 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 23:29:41.222668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214792 6208 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214795 6208 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214798 6208 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214801 6208 flags.go:64] FLAG: --runonce="false" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214804 6208 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214807 6208 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214810 6208 flags.go:64] FLAG: --seccomp-default="false" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214813 6208 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214816 6208 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214819 6208 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214823 6208 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214826 6208 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214829 6208 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214832 6208 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214835 6208 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214838 6208 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214841 6208 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214844 6208 flags.go:64] FLAG: --system-cgroups="" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214847 6208 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214852 6208 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214855 6208 flags.go:64] FLAG: --tls-cert-file="" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214858 6208 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214862 6208 flags.go:64] FLAG: --tls-min-version="" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214865 6208 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214868 6208 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 23:29:41.223659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214871 6208 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214876 6208 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214879 6208 flags.go:64] FLAG: --v="2" Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214884 6208 flags.go:64] FLAG: --version="false" Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214888 6208 flags.go:64] FLAG: --vmodule="" Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214892 6208 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.214896 6208 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215052 6208 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215058 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215061 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215064 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215067 6208 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215069 6208 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215072 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215075 6208 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215078 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215080 6208 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215083 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215087 6208 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215091 6208 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:29:41.224288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215094 6208 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215097 6208 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215099 6208 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215102 6208 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215106 6208 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215108 6208 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215111 6208 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215113 6208 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215116 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215119 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215122 6208 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215124 6208 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215127 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215132 6208 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215134 6208 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215137 6208 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215140 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215142 6208 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215145 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215150 6208 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:29:41.224786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215152 6208 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215155 6208 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215158 6208 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215161 6208 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215163 6208 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215166 6208 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215168 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215171 6208 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215174 6208 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215177 6208 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215179 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215182 6208 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215184 6208 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215187 6208 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215190 6208 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215192 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215195 6208 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215198 6208 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215200 6208 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:29:41.225363 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215203 6208 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215206 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215208 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215211 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215214 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215216 6208 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215221 6208 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215223 6208 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215226 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215228 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215231 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215234 6208 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215238 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215242 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215244 6208 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215247 6208 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215249 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215252 6208 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215255 6208 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215258 6208 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:29:41.225878 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215261 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215263 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215266 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215268 6208 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215271 6208 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215274 6208 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215276 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215279 6208 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215282 6208 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215285 6208 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215287 6208 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215290 6208 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215293 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215295 6208 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215298 6208 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215302 6208 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215306 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215309 6208 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215315 6208 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:29:41.226479 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215317 6208 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215320 6208 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215322 6208 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215325 6208 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215328 6208 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215332 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215335 6208 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215338 6208 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215341 6208 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215343 6208 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215346 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215348 6208 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215351 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215353 6208 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215356 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.215383 6208 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 23:29:41.227390 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215418 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215424 6208 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215428 6208 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215432 6208 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215435 6208 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215438 6208 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215441 6208 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215444 6208 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215447 6208 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215449 6208 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215452 6208 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215455 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215458 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215461 6208 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215463 6208 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215466 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215470 6208 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215473 6208 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215475 6208 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:29:41.228121 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215478 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215481 6208 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215484 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215487 6208 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215490 6208 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215493 6208 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215496 6208 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215499 6208 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215501 6208 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215504 6208 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215506 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215509 6208 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215512 6208 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215515 6208 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215518 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215520 6208 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215523 6208 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215526 6208 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215528 6208 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215531 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:29:41.228705 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215533 6208 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215536 6208 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215538 6208 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215541 6208 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215543 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215546 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215549 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215551 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215553 6208 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215558 6208 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215561 6208 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215563 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215566 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215569 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215572 6208 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215575 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215578 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215580 6208 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215583 6208 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215586 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:29:41.229346 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215588 6208 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215591 6208 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215594 6208 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215597 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215599 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215602 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215604 6208 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215607 6208 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215610 6208 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215612 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215615 6208 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215617 6208 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215620 6208 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215622 6208 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215625 6208 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215628 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215630 6208 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215633 6208 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215636 6208 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:29:41.229926 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215640 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215643 6208 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215647 6208 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215650 6208 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215652 6208 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215656 6208 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215658 6208 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215662 6208 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215665 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215667 6208 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215670 6208 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215672 6208 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215675 6208 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215678 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215681 6208 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215683 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215686 6208 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215689 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215691 6208 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:29:41.230419 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215694 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215696 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215699 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215702 6208 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215705 6208 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215707 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215710 6208 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215712 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.215715 6208 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.219522 6208 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.219539 6208 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.219558 6208 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.219563 6208 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219618 6208 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219623 6208 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219627 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219630 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219633 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219635 6208 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:29:41.230871 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219638 6208 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219641 6208 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219644 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219646 6208 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219649 6208 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219653 6208 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219657 6208 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219660 6208 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219663 6208 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219666 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219669 6208 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219672 6208 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219675 6208 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219678 6208 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219680 6208 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219683 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219686 6208 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219689 6208 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219691 6208 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:29:41.231333 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219694 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219697 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219700 6208 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219703 6208 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219712 6208 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219715 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219718 6208 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219721 6208 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219724 6208 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219726 6208 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219729 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219732 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219735 6208 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219737 6208 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219740 6208 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219742 6208 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219745 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219748 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219750 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:29:41.231786 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219753 6208 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219755 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219758 6208 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219761 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219765 6208 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219768 6208 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219771 6208 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219773 6208 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219776 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219779 6208 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219782 6208 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219785 6208 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219787 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219790 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219793 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219796 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219798 6208 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219801 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219804 6208 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:29:41.232501 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219807 6208 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219810 6208 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219813 6208 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219815 6208 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219818 6208 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219820 6208 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219823 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219826 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219828 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219831 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219834 6208 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219836 6208 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219839 6208 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219841 6208 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219844 6208 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219846 6208 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219849 6208 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219852 6208 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219854 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219857 6208 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:29:41.232957 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219859 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219862 6208 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219865 6208 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219868 6208 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219870 6208 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219873 6208 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219875 6208 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219878 6208 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219881 6208 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219884 6208 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219886 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219889 6208 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219892 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219895 6208 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219897 6208 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219900 6208 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219902 6208 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219905 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219907 6208 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219910 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:29:41.233455 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219913 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219915 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.219918 6208 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.219945 6208 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220121 6208 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220127 6208 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220129 6208 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220133 6208 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220136 6208 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220139 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220143 6208 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220145 6208 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220176 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220180 6208 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220184 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220188 6208 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:29:41.234005 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220193 6208 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220197 6208 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220201 6208 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220207 6208 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220213 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220217 6208 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220222 6208 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220226 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220230 6208 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220236 6208 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220240 6208 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220245 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220248 6208 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220252 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220257 6208 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220261 6208 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220266 6208 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220269 6208 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220272 6208 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220274 6208 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:29:41.234414 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220277 6208 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220280 6208 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220282 6208 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220285 6208 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220287 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220290 6208 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220293 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220296 6208 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220298 6208 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220301 6208 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220303 6208 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220306 6208 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220310 6208 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220314 6208 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220317 6208 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220320 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220322 6208 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220325 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220328 6208 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:29:41.234900 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220331 6208 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220334 6208 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220336 6208 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220340 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220343 6208 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220346 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220349 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220351 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220354 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220358 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220360 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220363 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220365 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220368 6208 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220371 6208 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220373 6208 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220376 6208 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220378 6208 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220381 6208 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220384 6208 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:29:41.235370 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220386 6208 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220389 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220392 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220394 6208 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220397 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220400 6208 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220402 6208 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220405 6208 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220407 6208 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220410 6208 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220412 6208 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220415 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220417 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220420 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220423 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220425 6208 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220428 6208 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220431 6208 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220433 6208 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220436 6208 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:29:41.235842 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220438 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220441 6208 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220443 6208 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220446 6208 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220448 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220451 6208 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220453 6208 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220456 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220458 6208 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220461 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220482 6208 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220485 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220488 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220491 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220493 6208 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.220526 6208 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 23:29:41.236372 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220564 6208 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220568 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220571 6208 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220574 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220577 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220579 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220582 6208 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220584 6208 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220586 6208 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220589 6208 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220591 6208 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220594 6208 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220597 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220600 6208 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220603 6208 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220606 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220608 6208 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220611 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220613 6208 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 23:29:41.236760 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220616 6208 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220618 6208 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220621 6208 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220624 6208 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220628 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220631 6208 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220634 6208 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220639 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220642 6208 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220644 6208 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220646 6208 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220649 6208 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220652 6208 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220654 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220657 6208 feature_gate.go:386] unrecognized feature gate: Example Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220660 6208 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220663 6208 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220666 6208 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220668 6208 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220671 6208 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 23:29:41.237242 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220674 6208 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220676 6208 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220679 6208 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220682 6208 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220684 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220687 6208 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220690 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220693 6208 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220696 6208 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220698 6208 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220701 6208 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220703 6208 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220707 6208 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220711 6208 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220715 6208 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220717 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220720 6208 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220723 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220725 6208 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 23:29:41.237758 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220729 6208 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220731 6208 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220734 6208 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220736 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220739 6208 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220742 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220745 6208 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220747 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220752 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220756 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220758 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220761 6208 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220763 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220766 6208 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220769 6208 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220771 6208 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220774 6208 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220777 6208 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220779 6208 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220782 6208 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 23:29:41.238239 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220785 6208 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220788 6208 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220791 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220793 6208 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220796 6208 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220799 6208 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220801 6208 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220804 6208 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220806 6208 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220809 6208 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220811 6208 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220814 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220817 6208 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220820 6208 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220822 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220825 6208 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220827 6208 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220830 6208 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220833 6208 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220835 6208 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 23:29:41.238719 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220839 6208 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220842 6208 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220844 6208 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220847 6208 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220851 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220853 6208 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220856 6208 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:41.220858 6208 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.220913 6208 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.221871 6208 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.221926 6208 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.222869 6208 server.go:1014] "Starting client certificate rotation" Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.223163 6208 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.223236 6208 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 23:23:28 +0000 UTC" deadline="2028-06-16 09:19:46.59005382 +0000 UTC" Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.223273 6208 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14169h50m5.366784716s" Nov 03 23:29:41.239247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.223903 6208 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 23:29:41.239615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.224966 6208 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 23:29:41.239615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.226100 6208 log.go:25] "Validated CRI v1 runtime API" Nov 03 23:29:41.239615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.228389 6208 log.go:25] "Validated CRI v1 image API" Nov 03 23:29:41.239615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.229164 6208 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 23:29:41.239615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.231290 6208 fs.go:135] Filesystem UUIDs: map[1ccfce98-0105-40fd-beef-5c933ccf6c4c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d2e7a6e0-e1db-40a8-9367-4a47d372df81:/dev/nvme0n1p3] Nov 03 23:29:41.239770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.231313 6208 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05d9ec6dd57613e3fff2b9fb708bae5276e1fe784c0366f884b9aa5c1c3cf757/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05d9ec6dd57613e3fff2b9fb708bae5276e1fe784c0366f884b9aa5c1c3cf757/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/245123bc0625af9257eaa466ddc1dda141b9828de38b07b4012547e5e1227a1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/245123bc0625af9257eaa466ddc1dda141b9828de38b07b4012547e5e1227a1b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38ce56ebdf0ca36e6f651f549321a729bcd596b712616053e74b8ca532d613c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38ce56ebdf0ca36e6f651f549321a729bcd596b712616053e74b8ca532d613c3/userdata/shm major:0 minor:381 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42b2b5c55361355087f672ab789b67964e6751c7865bb7f83508e3e5c6b44913/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42b2b5c55361355087f672ab789b67964e6751c7865bb7f83508e3e5c6b44913/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bab34e7a561fe8fd6885618b4468a66449cae1d4a0ec3b7b200b388fb02aee2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bab34e7a561fe8fd6885618b4468a66449cae1d4a0ec3b7b200b388fb02aee2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5145f3a64bc6602c4631ae64b6c7924ccbd8d44c072e136d017d1c02b774ca6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5145f3a64bc6602c4631ae64b6c7924ccbd8d44c072e136d017d1c02b774ca6a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60c64add717a314691a3a2a24ef2b8fef80afc6735929b6877642dbd4d2b8755/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60c64add717a314691a3a2a24ef2b8fef80afc6735929b6877642dbd4d2b8755/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6830b700af5c1f0d22a5fe8f6f634f824c75e8953fc35ca8f7fc4c596f7853ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6830b700af5c1f0d22a5fe8f6f634f824c75e8953fc35ca8f7fc4c596f7853ce/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6aa0c4423bf56f007209bb369aab4fb4973eb8a0f0a72d99532be0b9d0cd137b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6aa0c4423bf56f007209bb369aab4fb4973eb8a0f0a72d99532be0b9d0cd137b/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7222e28a9cd1391b3af0c5894d3f7be951ab2be85191b93cf47bda6e3bfa03e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7222e28a9cd1391b3af0c5894d3f7be951ab2be85191b93cf47bda6e3bfa03e5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/757121206c662fd3568284fc55097fefc9bb916b12fafdaf41b494e17e64351f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/757121206c662fd3568284fc55097fefc9bb916b12fafdaf41b494e17e64351f/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77f4c610c96d62b403facb014d4d9d5f2feca820f4fbfad17e5b3a31bef1bdd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77f4c610c96d62b403facb014d4d9d5f2feca820f4fbfad17e5b3a31bef1bdd3/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/924db680829930627e55d2ab35f80475fc4c9245d0ad2de017a27480b7ed32ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/924db680829930627e55d2ab35f80475fc4c9245d0ad2de017a27480b7ed32ba/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92c0ec3dbccb50c66767557717dd84cdb65d12e65edd70a84cd706b09493716a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92c0ec3dbccb50c66767557717dd84cdb65d12e65edd70a84cd706b09493716a/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5f3d49f7567843143cc40a2ee82a15d7349002d5fca297f2e2bd6c81a1d6eca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5f3d49f7567843143cc40a2ee82a15d7349002d5fca297f2e2bd6c81a1d6eca/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9cb916fd007a09e818c1897a867326f80e0ea794ccb9d87872b92702fa2f8f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9cb916fd007a09e818c1897a867326f80e0ea794ccb9d87872b92702fa2f8f6/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb7dca0071dac6810a7f1d35d443a582391df8ab813a348be5215b8cd99446fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb7dca0071dac6810a7f1d35d443a582391df8ab813a348be5215b8cd99446fb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bce424346a312c6e62eaa196c1114b1df85e40966486216b215ab9b8b02034af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bce424346a312c6e62eaa196c1114b1df85e40966486216b215ab9b8b02034af/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1e96e41f0a73939ebffa16f757381cf9c8b4ab31bad667c62795b5f36d4b1db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1e96e41f0a73939ebffa16f757381cf9c8b4ab31bad667c62795b5f36d4b1db/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e446a430ade0ce7fe1db989827c1ab118270009eddf69f384faa9a2877ae575c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e446a430ade0ce7fe1db989827c1ab118270009eddf69f384faa9a2877ae575c/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef745d51de9b1e8bdaf3a77891a4bc05097a77ef09c18527b8d5963f28649b4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef745d51de9b1e8bdaf3a77891a4bc05097a77ef09c18527b8d5963f28649b4b/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1b9403b98d9df347f27b95a3c4ecab5e1938f10d7e18b5af052c7ec9f80a4c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1b9403b98d9df347f27b95a3c4ecab5e1938f10d7e18b5af052c7ec9f80a4c5/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9d48ebd1c2d01314b0b19c3286be36a1a49f83fc58b8b79b5d0e5fbe2850c57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9d48ebd1c2d01314b0b19c3286be36a1a49f83fc58b8b79b5d0e5fbe2850c57/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17b0bd49-65e6-4393-88ba-b578c5eab313/volumes/kubernetes.io~projected/kube-api-access-9ht52:{mountpoint:/var/lib/kubelet/pods/17b0bd49-65e6-4393-88ba-b578c5eab313/volumes/kubernetes.io~projected/kube-api-access-9ht52 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17b0bd49-65e6-4393-88ba-b578c5eab313/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/17b0bd49-65e6-4393-88ba-b578c5eab313/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1855ee11-b7e8-49f7-bf9d-4439ac060f20/volumes/kubernetes.io~projected/kube-api-access-gqglk:{mountpoint:/var/lib/kubelet/pods/1855ee11-b7e8-49f7-bf9d-4439ac060f20/volumes/kubernetes.io~projected/kube-api-access-gqglk major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e545f4e-ce4d-4a60-ba33-74207fb40c48/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1e545f4e-ce4d-4a60-ba33-74207fb40c48/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/271a3854-3368-425b-9fcd-690738740bb8/volumes/kubernetes.io~projected/kube-api-access-cctzq:{mountpoint:/var/lib/kubelet/pods/271a3854-3368-425b-9fcd-690738740bb8/volumes/kubernetes.io~projected/kube-api-access-cctzq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bc833b9-89ed-4f1a-8dec-602fffd9c072/volumes/kubernetes.io~projected/kube-api-access-pt84d:{mountpoint:/var/lib/kubelet/pods/2bc833b9-89ed-4f1a-8dec-602fffd9c072/volumes/kubernetes.io~projected/kube-api-access-pt84d major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bc833b9-89ed-4f1a-8dec-602fffd9c072/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/2bc833b9-89ed-4f1a-8dec-602fffd9c072/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34cd9f7a-0794-465b-9443-d6723c5505fb/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/34cd9f7a-0794-465b-9443-d6723c5505fb/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34cd9f7a-0794-465b-9443-d6723c5505fb/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/34cd9f7a-0794-465b-9443-d6723c5505fb/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34cd9f7a-0794-465b-9443-d6723c5505fb/volumes/kubernetes.io~projected/kube-api-access-m24tz:{mountpoint:/var/lib/kubelet/pods/34cd9f7a-0794-465b-9443-d6723c5505fb/volumes/kubernetes.io~projected/kube-api-access-m24tz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36489906-6e8f-4aa6-ae96-f872732808ed/volumes/kubernetes.io~projected/kube-api-access-7kkst:{mountpoint:/var/lib/kubelet/pods/36489906-6e8f-4aa6-ae96-f872732808ed/volumes/kubernetes.io~projected/kube-api-access-7kkst major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37f77e6f-4bd8-4a9e-ab23-c1337365cac1/volumes/kubernetes.io~projected/kube-api-access-kp7xk:{mountpoint:/var/lib/kubelet/pods/37f77e6f-4bd8-4a9e-ab23-c1337365cac1/volumes/kubernetes.io~projected/kube-api-access-kp7xk major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37f77e6f-4bd8-4a9e-ab23-c1337365cac1/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/37f77e6f-4bd8-4a9e-ab23-c1337365cac1/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37f77e6f-4bd8-4a9e-ab23-c1337365cac1/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/37f77e6f-4bd8-4a9e-ab23-c1337365cac1/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~projected/kube-api-access-9h2zm:{mountpoint:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~projected/kube-api-access-9h2zm major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32/volumes/kubernetes.io~projected/kube-api-access-5c46g:{mountpoint:/var/lib/kubelet/pods/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32/volumes/kubernetes.io~projected/kube-api-access-5c46g major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72d93d4f-2af9-44d5-b8e5-542b30760f34/volumes/kubernetes.io~projected/kube-api-access-kr6f7:{mountpoint:/var/lib/kubelet/pods/72d93d4f-2af9-44d5-b8e5-542b30760f34/volumes/kubernetes.io~projected/kube-api-access-kr6f7 major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c784845-c509-4b51-86b8-ce9075600208/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9c784845-c509-4b51-86b8-ce9075600208/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fc8829d-b25c-4952-8cd8-816aa35089ac/volumes/kubernetes.io~projected/kube-api-access-9bj29:{mountpoint:/var/lib/kubelet/pods/9fc8829d-b25c-4952-8cd8-816aa35089ac/volumes/kubernetes.io~projected/kube-api-access-9bj29 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fc8829d-b25c-4952-8cd8-816aa35089ac/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/9fc8829d-b25c-4952-8cd8-816aa35089ac/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes/kubernetes.io~projected/kube-api-access-lfzcx:{mountpoint:/var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes/kubernetes.io~projected/kube-api-access-lfzcx major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ceed0dfc-9a74-469f-a91d-aab8c3fa402d/volumes/kubernetes.io~projected/kube-api-access-rj5tv:{mountpoint:/var/lib/kubelet/pods/ceed0dfc-9a74-469f-a91d-aab8c3fa402d/volumes/kubernetes.io~projected/kube-api-access-rj5tv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes/kubernetes.io~projected/kube-api-access-76cz7:{mountpoint:/var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes/kubernetes.io~projected/kube-api-access-76cz7 major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc219915-7ac3-4e9b-863f-4c641a8b6540/volumes/kubernetes.io~projected/kube-api-access-j8wtn:{mountpoint:/var/lib/kubelet/pods/dc219915-7ac3-4e9b-863f-4c641a8b6540/volumes/kubernetes.io~projected/kube-api-access-j8wtn major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f42ad3cc-de80-43bb-b548-dfa6c78618e5/volumes/kubernetes.io~projected/kube-api-access-sdfj2:{mountpoint:/var/lib/kubelet/pods/f42ad3cc-de80-43bb-b548-dfa6c78618e5/volumes/kubernetes.io~projected/kube-api-access-sdfj2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f42ad3cc-de80-43bb-b548-dfa6c78618e5/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f42ad3cc-de80-43bb-b548-dfa6c78618e5/volumes/kubernetes.io~secret/metrics-certs major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes/kubernetes.io~projected/kube-api-access-z44cj:{mountpoint:/var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes/kubernetes.io~projected/kube-api-access-z44cj major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volume-subpaths/web-config/config-reloader/2 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volume-subpaths/web-config/prometheus/15 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~projected/kube-api-access-qqnbk:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~projected/kube-api-access-qqnbk major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~projected/tls-assets major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/config major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/web-config major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdc30eef-1cd6-449c-9aef-d15732f1a2f8/volumes/kubernetes.io~projected/kube-api-access-8wfwh:{mountpoint:/var/lib/kubelet/pods/fdc30eef-1cd6-449c-9aef-d15732f1a2f8/volumes/kubernetes.io~projected/kube-api-access-8wfwh major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdc30eef-1cd6-449c-9aef-d15732f1a2f8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fdc30eef-1cd6-449c-9aef-d15732f1a2f8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdc30eef-1cd6-449c-9aef-d15732f1a2f8/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/fdc30eef-1cd6-449c-9aef-d15732f1a2f8/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fecf1197-13f8-4c4a-a878-c784e18385d6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fecf1197-13f8-4c4a-a878-c784e18385d6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fecf1197-13f8-4c4a-a878-c784e18385d6/volumes/kubernetes.io~projected/kube-api-access-v4gbd:{mountpoint:/var/lib/kubelet/pods/fecf1197-13f8-4c4a-a878-c784e18385d6/volumes/kubernetes.io~projected/kube-api-access-v4gbd major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fecf1197-13f8-4c4a-a878-c784e18385d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fecf1197-13f8-4c4a-a878-c784e18385d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4ac1ec12ee81b2cb77fdb4da2c6aa3f6a48692a4dc3871127482caaebd7e4fba/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e7dcdfd6698bfd90aa7479bb2453c9321d37c2cf45fe67f2d723b78834055467/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c01a7f1a202bc8fd205800ee67e4c1bcf34429428ea8eb52d04f5d7a8047367d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/646b943574428be73cb47ba9103fc57d7c975cbb85d4ffed8a19df6c5a5e442b/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/c1880f0f72b49dd5cea6ce5d8560309743a768df603fa9e36dda3628a35edf1a/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a272e06acde482610199395d551e4290fdc4e103457003ad980ccf4299ef15f5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7acbbf790a51ae9c617486bdf64e79c7e62b9da41a42cc7500bf9eb98d4f6edf/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b6538c29ce0d65f2796ca1dd5f1ba1d78855577118118ea083e630bc39b6d07f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/06fb71e563fa00c574ca8781ba1b10ebfcd6088656a9871982164107ba9d6df2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ccdc91b845426a79a76db23e6f39d1463fbc347961326a9975fb8a4ab18a1ba1/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/d31e13c22a020ceec15f3c5306c56ee2ec4433acd2c710e2b6db0ab54f5edc07/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/bc85fed10795e258035e8d1c6b22f235b5f7085374c408edb79f4bdaef725bee/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8e0697961feb1a0dc3aba2a870d63e7b99a1ead4e81939cdc0ffca559cc56872/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b2639f118387233e8e34e35cb443de7e7cacd0c1ce2baba3ac642c9868324102/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/4f07893c6e4858668847b8292a31629a72107305cf7435f09d4ad4854f188469/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6faa3a6a2c318be91292b30d8d71fa37e63d4c53a0441016f7e33de4e334d86d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/319d9be2ef369f60268763987ddd8d40d506dfebbd6bdf5d0de5cb512e0c96ce/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/07837e01f78660c0583697c188731a1206e831c2df8a9a50bf29fa5596b6d358/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/859ff4f0863fb7034f38254fdfa4ef08c20956b6b446508be166e590c57387e6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/64c20e97f3029f7f4345c965993a369103137a514cf0c98cf0d0bfe5c9175fd4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/270c7820fbcde0b92e8dcbb63b6d8ea89989f9edb3440fdd6335aa7ebfce3519/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c7825d190692086a0cd9a54684f70d778f4a7af345c261874d8a27864e132305/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/9070c7ac4c0c8f857ca0cfe13f5bebcc6f2a379cbd8905a2304a88dbf32402eb/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/0f8e11b10b4d8d46a6f5e72a20857daf1d9050d5fda0cd96ed57061739b9a41b/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/aef1be8419a9b0c500e8caa9fb25d0c167e9603cb23d5fdead1804b225975fc3/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/d888eecd945d0d9283985b3a9e8788bd12c5ea9dabf132dfd7b44896d2fd62c8/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/03efb8886eb94199d00b71b8085e7e8cdac5c4b4ff0f9c76d443298ae5846ead/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/a6ee673bbde469d97cb3b40c337df1bc7675840afc3c8dd6d87d94c365db26fc/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/e20dc9bb490eed30a41922182f09f94d64d6c2dd1f44d83034eff99de1609d8b/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/6d2693a08b1f6e2bf9bb8b169e970571a15607ed5ff778b2b4040dde0414f5b7/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/fb243095c034e1047513ab65d67efda1c433f74abb8a941011a214da8bcc82db/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/590d07f25df2da9fe7ca642bf3af4c5c9614fca651072d3c2bd6f61e40834f37/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/4f6e6a6d2a1db13135c35ab5a3116e0c53186da99a2b6a976a39383703b8c7e9/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/3391bae821b77a0e5db61194afec9b16e743c4661fa563c1a566ad3d0975d284/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/477c11cb19b21a46529b85ee6c658a87c3ac2eeaace852e2b5bdc19dce25785a/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/594a211efb0dd1c501cfbd131c586a6f6728782eba9c5e0594877675e1ec34f6/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/43061571773ec401258961ab8f185df09090df257362b937cbe41b37494e2680/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/03581c6910f797b1f712cca6fd2f52339ca0657a2d10b5a5bff17da44488a256/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/c20ccfe8fb9e14bda0f59f475cbcdce89ee8db77cd5a329c1ee9c1cf622c406e/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/8a1254d41f4af6790120a3d52d2cb7b9523cb475d1f59f7a619ea5c22d82669d/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/8664ffe4f80e4fef477bce50198e294b5be3804c3433eada1d6f94b02754f235/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/1082cf6f45fb6caae855120c465497d62b7d8a8bc7a1cdf14ac2e360677da8e9/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/96f83bd00203c9ac696d096ca6613444f82b29cd35c6c189583ad2c74ee02a74/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/b90ce3710278f0e785053163da706f46f9dc5b5770ae23212fb383920a6d115f/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/058943555b5abea882d31f9f20c04b8e15adf5c01001dc55731d8232ac6617a6/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/14ffd6c54f5347ee9a8de972a110ffd10c4c6ee96549502174ef0c8d9a1cc2f4/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/5a891b29b08ae23a46a00b59caa793245fff56b7b5baa74cab7c31170ed3b3d9/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/e2099df976426d96831cfff1e6096d44bba434bd2b0780ce0c5d96bf9ee3eeea/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/e1319e9d77cc4d3d91d11060276550e71b92152d960a190144a55d1a8f36f1d3/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/bdd1c63f9b886a76c3d5230d23a72f33b9754648de0f289ce9f78577dacf4339/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/9e55646994896b27758a6b8135335d7c9c0b48894c4af168429050360382cbbc/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/b9527a176b51543a479dc1ad7b2288eff348a06bb1c3868e3b7ef438d8245fae/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/a60be6eb3eba3a1bde719066018ad1b3b30affec8ec01e78b7d4485fdc2e797a/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/e384cf30395b4c4ee31411537c8e8312e9cec2e599f5d49bd54146a1dfac3788/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/3c09e6ab4c7fed2584caefc0ca6499b1dd9944a3bcc54004a804cf85bca96ad8/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/cf7118ec6fd8e119f8dfd9b0bdebe7a3e226ed679dff4b2ed4c5f9150a210ec1/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/580af220c886dfa4ba51a07951a6420a5f9c856a552601170fedc3dc4fc93d79/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/59d0aba4f5d93e76bbf9afdc253414afc694543905d43836b0254f6bdd3a82f7/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/310177a8254d6bd85160cfc97bd1347406359ca61532b4fc6bf0337d305b649f/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/7346c4c14021cbce74f0f61aaf0c1d4cb86d0396f0ce6bb167ccbac730ed2db2/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/632181f2525b741df93037d03ec5cd1bdcb7ad54e4bb3e7fea5ab54d9d8931ed/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/4808f48b0887537b2ed2cf2a1f5f74ee7818f7364db6db3b8870e33763d121a8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/301c7ddc3db7bf54dd5be098eb9c5cdeae723668786d0124ae0e3405931fe7ff/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/774f7cc4f0e04284c17abf6741db5fba8ca54e879910e6ef4942590b58f94930/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c40954ff32e01a0f7dcc2f2b46f1a910f40bccb5bc87e19a05aac0409e854055/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/40f2b176039ed7c64707d8b0173e7594f8bfbc34514566c5b2043d48edf4dc54/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/8ce388ce1d956be091ddeca5b7a7f7d39141a062f9bf8daa8f0040e7b5da95fc/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/989485ce2b25abd7d1dbc77a9e36a31950817dd01b5fbbf9fbb6c3a77d96dff3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/37d2241112439d754d1b601124cea664bb89fec28b1f0b82c78be8de63ed4e4a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7a99de75cc61ca18fbeacd497e7be02adb40e6c138db1d5770bd425bb0f31ee3/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 23:29:41.240019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.239241 6208 manager.go:217] Machine: {Timestamp:2026-11-03 23:29:41.238751997 +0000 UTC m=+0.061348150 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100133 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e8496ce9b0f60757912438e752996 SystemUUID:ec2e8496-ce9b-0f60-7579-12438e752996 BootID:61da4c6f-96a0-4144-b92d-a9d78058494b Filesystems:[{Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f42ad3cc-de80-43bb-b548-dfa6c78618e5/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes/kubernetes.io~projected/kube-api-access-lfzcx DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/37f77e6f-4bd8-4a9e-ab23-c1337365cac1/volumes/kubernetes.io~projected/kube-api-access-kp7xk DeviceMajor:0 DeviceMinor:243 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9cb916fd007a09e818c1897a867326f80e0ea794ccb9d87872b92702fa2f8f6/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5145f3a64bc6602c4631ae64b6c7924ccbd8d44c072e136d017d1c02b774ca6a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdc30eef-1cd6-449c-9aef-d15732f1a2f8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/72d93d4f-2af9-44d5-b8e5-542b30760f34/volumes/kubernetes.io~projected/kube-api-access-kr6f7 DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fecf1197-13f8-4c4a-a878-c784e18385d6/volumes/kubernetes.io~projected/kube-api-access-v4gbd DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes/kubernetes.io~projected/kube-api-access-76cz7 DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34cd9f7a-0794-465b-9443-d6723c5505fb/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34cd9f7a-0794-465b-9443-d6723c5505fb/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:311 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38ce56ebdf0ca36e6f651f549321a729bcd596b712616053e74b8ca532d613c3/userdata/shm DeviceMajor:0 DeviceMinor:381 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32/volumes/kubernetes.io~projected/kube-api-access-5c46g DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37f77e6f-4bd8-4a9e-ab23-c1337365cac1/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05d9ec6dd57613e3fff2b9fb708bae5276e1fe784c0366f884b9aa5c1c3cf757/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17b0bd49-65e6-4393-88ba-b578c5eab313/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fecf1197-13f8-4c4a-a878-c784e18385d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ceed0dfc-9a74-469f-a91d-aab8c3fa402d/volumes/kubernetes.io~projected/kube-api-access-rj5tv DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bab34e7a561fe8fd6885618b4468a66449cae1d4a0ec3b7b200b388fb02aee2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc219915-7ac3-4e9b-863f-4c641a8b6540/volumes/kubernetes.io~projected/kube-api-access-j8wtn DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1b9403b98d9df347f27b95a3c4ecab5e1938f10d7e18b5af052c7ec9f80a4c5/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:378 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bce424346a312c6e62eaa196c1114b1df85e40966486216b215ab9b8b02034af/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~projected/kube-api-access-9h2zm DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes/kubernetes.io~projected/kube-api-access-z44cj DeviceMajor:0 DeviceMinor:380 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fecf1197-13f8-4c4a-a878-c784e18385d6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2bc833b9-89ed-4f1a-8dec-602fffd9c072/volumes/kubernetes.io~projected/kube-api-access-pt84d DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60c64add717a314691a3a2a24ef2b8fef80afc6735929b6877642dbd4d2b8755/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37f77e6f-4bd8-4a9e-ab23-c1337365cac1/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:257 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36489906-6e8f-4aa6-ae96-f872732808ed/volumes/kubernetes.io~projected/kube-api-access-7kkst DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb7dca0071dac6810a7f1d35d443a582391df8ab813a348be5215b8cd99446fb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdc30eef-1cd6-449c-9aef-d15732f1a2f8/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9d48ebd1c2d01314b0b19c3286be36a1a49f83fc58b8b79b5d0e5fbe2850c57/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42b2b5c55361355087f672ab789b67964e6751c7865bb7f83508e3e5c6b44913/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92c0ec3dbccb50c66767557717dd84cdb65d12e65edd70a84cd706b09493716a/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6aa0c4423bf56f007209bb369aab4fb4973eb8a0f0a72d99532be0b9d0cd137b/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~projected/kube-api-access-qqnbk DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1855ee11-b7e8-49f7-bf9d-4439ac060f20/volumes/kubernetes.io~projected/kube-api-access-gqglk DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/271a3854-3368-425b-9fcd-690738740bb8/volumes/kubernetes.io~projected/kube-api-access-cctzq DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/245123bc0625af9257eaa466ddc1dda141b9828de38b07b4012547e5e1227a1b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9fc8829d-b25c-4952-8cd8-816aa35089ac/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9fc8829d-b25c-4952-8cd8-816aa35089ac/volumes/kubernetes.io~projected/kube-api-access-9bj29 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7222e28a9cd1391b3af0c5894d3f7be951ab2be85191b93cf47bda6e3bfa03e5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2bc833b9-89ed-4f1a-8dec-602fffd9c072/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17b0bd49-65e6-4393-88ba-b578c5eab313/volumes/kubernetes.io~projected/kube-api-access-9ht52 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77f4c610c96d62b403facb014d4d9d5f2feca820f4fbfad17e5b3a31bef1bdd3/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/924db680829930627e55d2ab35f80475fc4c9245d0ad2de017a27480b7ed32ba/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fdc30eef-1cd6-449c-9aef-d15732f1a2f8/volumes/kubernetes.io~projected/kube-api-access-8wfwh DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e446a430ade0ce7fe1db989827c1ab118270009eddf69f384faa9a2877ae575c/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a95c441-120e-44ce-8b68-516030f94a91/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:379 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/757121206c662fd3568284fc55097fefc9bb916b12fafdaf41b494e17e64351f/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5f3d49f7567843143cc40a2ee82a15d7349002d5fca297f2e2bd6c81a1d6eca/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1e96e41f0a73939ebffa16f757381cf9c8b4ab31bad667c62795b5f36d4b1db/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f42ad3cc-de80-43bb-b548-dfa6c78618e5/volumes/kubernetes.io~projected/kube-api-access-sdfj2 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34cd9f7a-0794-465b-9443-d6723c5505fb/volumes/kubernetes.io~projected/kube-api-access-m24tz DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef745d51de9b1e8bdaf3a77891a4bc05097a77ef09c18527b8d5963f28649b4b/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e545f4e-ce4d-4a60-ba33-74207fb40c48/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c784845-c509-4b51-86b8-ce9075600208/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6830b700af5c1f0d22a5fe8f6f634f824c75e8953fc35ca8f7fc4c596f7853ce/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:38ce56ebdf0ca36 MacAddress:b2:47:fe:39:a0:cb Speed:10000 Mtu:8901} {Name:60c64add717a314 MacAddress:1a:53:d6:df:39:6b Speed:10000 Mtu:8901} {Name:757121206c662fd MacAddress:a6:94:47:95:ab:96 Speed:10000 Mtu:8901} {Name:924db6808299306 MacAddress:06:2f:7c:84:7b:ea Speed:10000 Mtu:8901} {Name:92c0ec3dbccb50c MacAddress:d2:bf:f9:a5:ff:68 Speed:10000 Mtu:8901} {Name:b5f3d49f7567843 MacAddress:9a:bf:27:ba:3f:cb Speed:10000 Mtu:8901} {Name:b9cb916fd007a09 MacAddress:2a:78:aa:f8:41:35 Speed:10000 Mtu:8901} {Name:bce424346a312c6 MacAddress:52:d0:1d:96:50:11 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d7:e9:a7:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:62:0f:c4:22:17:e0 Speed:0 Mtu:8901} {Name:e1e96e41f0a7393 MacAddress:e2:4f:f2:ff:fb:12 Speed:10000 Mtu:8901} {Name:e446a430ade0ce7 MacAddress:c2:b6:20:7a:b8:a2 Speed:10000 Mtu:8901} {Name:ef745d51de9b1e8 MacAddress:5e:6c:4e:7c:f8:ce Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d7:e9:a7:55 Speed:0 Mtu:9001} {Name:f1b9403b98d9df3 MacAddress:fe:13:70:d1:57:5c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:d6:d4:d5:ef:10:0a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:ad:0b:67:a0:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 23:29:41.240019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.239599 6208 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 23:29:41.240019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.239645 6208 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 23:29:41.240019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.239910 6208 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 23:29:41.240414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.239932 6208 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-137.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 23:29:41.240414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240099 6208 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 23:29:41.240414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240107 6208 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 23:29:41.240414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240121 6208 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 23:29:41.240414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240134 6208 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 23:29:41.240414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240209 6208 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 23:29:41.240414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240366 6208 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 23:29:41.240641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240464 6208 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 23:29:41.240641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240498 6208 kubelet.go:493] "Attempting to sync node with API server" Nov 03 23:29:41.240641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240507 6208 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 23:29:41.240641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240517 6208 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 23:29:41.240641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240526 6208 kubelet.go:405] "Adding apiserver pod source" Nov 03 23:29:41.240641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.240534 6208 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 23:29:41.242792 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.242383 6208 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 23:29:41.242792 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.242404 6208 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 23:29:41.243100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.243081 6208 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 23:29:41.246009 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.245994 6208 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 23:29:41.246009 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246009 6208 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 23:29:41.246136 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246125 6208 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 23:29:41.246323 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246312 6208 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 23:29:41.246353 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246342 6208 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 23:29:41.246435 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246426 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 23:29:41.246465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246441 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 23:29:41.246465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246447 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 23:29:41.246465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246452 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 23:29:41.246465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246458 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 23:29:41.246465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246464 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 23:29:41.246607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246469 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 23:29:41.246607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246475 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 23:29:41.246607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246483 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 23:29:41.246607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246489 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 23:29:41.246607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246498 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 23:29:41.246607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246508 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 23:29:41.246607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246535 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 23:29:41.246607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.246540 6208 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 23:29:41.250132 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.250118 6208 server.go:1263] "Started kubelet" Nov 03 23:29:41.250187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.250164 6208 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 23:29:41.250556 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.250308 6208 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 23:29:41.250556 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.250383 6208 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 23:29:41.250670 ip-10-0-138-137 systemd[1]: Started Kubernetes Kubelet. Nov 03 23:29:41.250874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.250857 6208 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 23:29:41.251979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.251953 6208 server.go:317] "Adding debug handlers to kubelet server" Nov 03 23:29:41.254261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.253689 6208 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 23:29:41.254261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.253726 6208 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 23:29:41.254261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.253795 6208 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 23:23:30 +0000 UTC" deadline="2028-07-21 22:26:42.95554531 +0000 UTC" Nov 03 23:29:41.254261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.253834 6208 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15022h57m1.701714999s" Nov 03 23:29:41.254261 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:29:41.253872 6208 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-137.us-east-2.compute.internal\" not found" Nov 03 23:29:41.254711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.254374 6208 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 23:29:41.254711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.254387 6208 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 23:29:41.254711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.254418 6208 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 23:29:41.257464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.256711 6208 factory.go:55] Registering systemd factory Nov 03 23:29:41.257464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.256742 6208 factory.go:223] Registration of the systemd container factory successfully Nov 03 23:29:41.257464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.257108 6208 factory.go:153] Registering CRI-O factory Nov 03 23:29:41.257464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.257134 6208 factory.go:223] Registration of the crio container factory successfully Nov 03 23:29:41.257464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.257188 6208 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 23:29:41.257464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.257209 6208 factory.go:103] Registering Raw factory Nov 03 23:29:41.257464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.257227 6208 manager.go:1199] Started watching for new ooms in manager Nov 03 23:29:41.259728 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.257707 6208 manager.go:319] Starting recovery of all containers Nov 03 23:29:41.262248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262141 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53faae9-d5e5-4c53-8f4c-1f421e40f203" volumeName="kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle" seLinuxMountContext="" Nov 03 23:29:41.262248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262195 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53faae9-d5e5-4c53-8f4c-1f421e40f203" volumeName="kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj" seLinuxMountContext="" Nov 03 23:29:41.262248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262213 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bc833b9-89ed-4f1a-8dec-602fffd9c072" volumeName="kubernetes.io/empty-dir/2bc833b9-89ed-4f1a-8dec-602fffd9c072-tmp-dir" seLinuxMountContext="" Nov 03 23:29:41.262248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262226 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a95c441-120e-44ce-8b68-516030f94a91" volumeName="kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-metrics-client-ca" seLinuxMountContext="" Nov 03 23:29:41.262248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262241 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a95c441-120e-44ce-8b68-516030f94a91" volumeName="kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-client-tls" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262255 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a707ad-9256-4eb1-bf0f-eae7ab6c8d32" volumeName="kubernetes.io/configmap/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-serviceca" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262268 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" volumeName="kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262282 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262299 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fecf1197-13f8-4c4a-a878-c784e18385d6" volumeName="kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-env-overrides" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262313 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fecf1197-13f8-4c4a-a878-c784e18385d6" volumeName="kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-config" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262322 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1855ee11-b7e8-49f7-bf9d-4439ac060f20" volumeName="kubernetes.io/empty-dir/1855ee11-b7e8-49f7-bf9d-4439ac060f20-tmp-dir" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262331 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bc833b9-89ed-4f1a-8dec-602fffd9c072" volumeName="kubernetes.io/secret/2bc833b9-89ed-4f1a-8dec-602fffd9c072-metrics-tls" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262338 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fecf1197-13f8-4c4a-a878-c784e18385d6" volumeName="kubernetes.io/projected/fecf1197-13f8-4c4a-a878-c784e18385d6-kube-api-access-v4gbd" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262346 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17b0bd49-65e6-4393-88ba-b578c5eab313" volumeName="kubernetes.io/projected/17b0bd49-65e6-4393-88ba-b578c5eab313-kube-api-access-9ht52" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262356 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" volumeName="kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262364 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53faae9-d5e5-4c53-8f4c-1f421e40f203" volumeName="kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262372 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53faae9-d5e5-4c53-8f4c-1f421e40f203" volumeName="kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262379 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262386 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262394 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e545f4e-ce4d-4a60-ba33-74207fb40c48" volumeName="kubernetes.io/secret/1e545f4e-ce4d-4a60-ba33-74207fb40c48-agent-certs" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262401 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a95c441-120e-44ce-8b68-516030f94a91" volumeName="kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262408 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc219915-7ac3-4e9b-863f-4c641a8b6540" volumeName="kubernetes.io/projected/dc219915-7ac3-4e9b-863f-4c641a8b6540-kube-api-access-j8wtn" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262426 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53faae9-d5e5-4c53-8f4c-1f421e40f203" volumeName="kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262434 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53faae9-d5e5-4c53-8f4c-1f421e40f203" volumeName="kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262442 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262463 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fecf1197-13f8-4c4a-a878-c784e18385d6" volumeName="kubernetes.io/secret/fecf1197-13f8-4c4a-a878-c784e18385d6-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262471 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e545f4e-ce4d-4a60-ba33-74207fb40c48" volumeName="kubernetes.io/configmap/1e545f4e-ce4d-4a60-ba33-74207fb40c48-konnectivity-ca" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262480 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bc833b9-89ed-4f1a-8dec-602fffd9c072" volumeName="kubernetes.io/configmap/2bc833b9-89ed-4f1a-8dec-602fffd9c072-config-volume" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262490 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37f77e6f-4bd8-4a9e-ab23-c1337365cac1" volumeName="kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262498 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" volumeName="kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262512 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" volumeName="kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262520 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" volumeName="kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262528 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262535 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262543 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72d93d4f-2af9-44d5-b8e5-542b30760f34" volumeName="kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262550 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262558 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262566 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262574 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc30eef-1cd6-449c-9aef-d15732f1a2f8" volumeName="kubernetes.io/empty-dir/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-textfile" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262584 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc30eef-1cd6-449c-9aef-d15732f1a2f8" volumeName="kubernetes.io/projected/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-kube-api-access-8wfwh" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262591 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc30eef-1cd6-449c-9aef-d15732f1a2f8" volumeName="kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-tls" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262599 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc30eef-1cd6-449c-9aef-d15732f1a2f8" volumeName="kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262606 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36489906-6e8f-4aa6-ae96-f872732808ed" volumeName="kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-cni-binary-copy" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262615 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36489906-6e8f-4aa6-ae96-f872732808ed" volumeName="kubernetes.io/projected/36489906-6e8f-4aa6-ae96-f872732808ed-kube-api-access-7kkst" seLinuxMountContext="" Nov 03 23:29:41.262603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262630 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" volumeName="kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262637 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53faae9-d5e5-4c53-8f4c-1f421e40f203" volumeName="kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262645 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262653 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc30eef-1cd6-449c-9aef-d15732f1a2f8" volumeName="kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262660 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fecf1197-13f8-4c4a-a878-c784e18385d6" volumeName="kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-script-lib" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262668 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37f77e6f-4bd8-4a9e-ab23-c1337365cac1" volumeName="kubernetes.io/configmap/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-metrics-client-ca" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262676 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" volumeName="kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262684 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" volumeName="kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262693 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262700 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36489906-6e8f-4aa6-ae96-f872732808ed" volumeName="kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-multus-daemon-config" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262707 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a95c441-120e-44ce-8b68-516030f94a91" volumeName="kubernetes.io/projected/3a95c441-120e-44ce-8b68-516030f94a91-kube-api-access-9h2zm" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262716 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" volumeName="kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262728 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262735 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262743 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="271a3854-3368-425b-9fcd-690738740bb8" volumeName="kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-binary-copy" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262750 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" volumeName="kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262758 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" volumeName="kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262766 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ceed0dfc-9a74-469f-a91d-aab8c3fa402d" volumeName="kubernetes.io/projected/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kube-api-access-rj5tv" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262774 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" volumeName="kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262781 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f42ad3cc-de80-43bb-b548-dfa6c78618e5" volumeName="kubernetes.io/projected/f42ad3cc-de80-43bb-b548-dfa6c78618e5-kube-api-access-sdfj2" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262788 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc30eef-1cd6-449c-9aef-d15732f1a2f8" volumeName="kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-metrics-client-ca" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262796 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1855ee11-b7e8-49f7-bf9d-4439ac060f20" volumeName="kubernetes.io/projected/1855ee11-b7e8-49f7-bf9d-4439ac060f20-kube-api-access-gqglk" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262803 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37f77e6f-4bd8-4a9e-ab23-c1337365cac1" volumeName="kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-tls" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262814 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c784845-c509-4b51-86b8-ce9075600208" volumeName="kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262823 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262834 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262845 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="271a3854-3368-425b-9fcd-690738740bb8" volumeName="kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262857 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34cd9f7a-0794-465b-9443-d6723c5505fb" volumeName="kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-tuned" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262869 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fc8829d-b25c-4952-8cd8-816aa35089ac" volumeName="kubernetes.io/projected/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-api-access-9bj29" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262882 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" volumeName="kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262893 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262904 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262915 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="271a3854-3368-425b-9fcd-690738740bb8" volumeName="kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262929 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" volumeName="kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262946 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc219915-7ac3-4e9b-863f-4c641a8b6540" volumeName="kubernetes.io/configmap/dc219915-7ac3-4e9b-863f-4c641a8b6540-iptables-alerter-script" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262960 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bc833b9-89ed-4f1a-8dec-602fffd9c072" volumeName="kubernetes.io/projected/2bc833b9-89ed-4f1a-8dec-602fffd9c072-kube-api-access-pt84d" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262972 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34cd9f7a-0794-465b-9443-d6723c5505fb" volumeName="kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-tmp" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262984 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37f77e6f-4bd8-4a9e-ab23-c1337365cac1" volumeName="kubernetes.io/projected/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-kube-api-access-kp7xk" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.262995 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" volumeName="kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263006 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f42ad3cc-de80-43bb-b548-dfa6c78618e5" volumeName="kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263018 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53faae9-d5e5-4c53-8f4c-1f421e40f203" volumeName="kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263098 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263113 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263124 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34cd9f7a-0794-465b-9443-d6723c5505fb" volumeName="kubernetes.io/projected/34cd9f7a-0794-465b-9443-d6723c5505fb-kube-api-access-m24tz" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263137 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a95c441-120e-44ce-8b68-516030f94a91" volumeName="kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263148 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a95c441-120e-44ce-8b68-516030f94a91" volumeName="kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-federate-client-tls" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263159 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a95c441-120e-44ce-8b68-516030f94a91" volumeName="kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263173 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a707ad-9256-4eb1-bf0f-eae7ab6c8d32" volumeName="kubernetes.io/projected/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-kube-api-access-5c46g" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263184 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fc8829d-b25c-4952-8cd8-816aa35089ac" volumeName="kubernetes.io/empty-dir/9fc8829d-b25c-4952-8cd8-816aa35089ac-data-volume" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263196 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fc8829d-b25c-4952-8cd8-816aa35089ac" volumeName="kubernetes.io/secret/9fc8829d-b25c-4952-8cd8-816aa35089ac-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263210 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f866b162-3fe8-4a69-9ae0-387e0c18d16d" volumeName="kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263222 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17b0bd49-65e6-4393-88ba-b578c5eab313" volumeName="kubernetes.io/secret/17b0bd49-65e6-4393-88ba-b578c5eab313-cert" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263236 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="271a3854-3368-425b-9fcd-690738740bb8" volumeName="kubernetes.io/projected/271a3854-3368-425b-9fcd-690738740bb8-kube-api-access-cctzq" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263249 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a95c441-120e-44ce-8b68-516030f94a91" volumeName="kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263261 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fc8829d-b25c-4952-8cd8-816aa35089ac" volumeName="kubernetes.io/configmap/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263272 6208 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" volumeName="kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert" seLinuxMountContext="" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263287 6208 reconstruct.go:97] "Volume reconstruction finished" Nov 03 23:29:41.264320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.263294 6208 reconciler.go:29] "Reconciler: start to sync state" Nov 03 23:29:41.266479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.264824 6208 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 23:29:41.269852 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.269822 6208 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 23:29:41.270842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.270821 6208 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 23:29:41.271976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.271958 6208 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 23:29:41.273898 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.273490 6208 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 23:29:41.286407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.286373 6208 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 23:29:41.286407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.286406 6208 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 23:29:41.286617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.286428 6208 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 23:29:41.286617 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:29:41.286468 6208 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 23:29:41.294368 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.294341 6208 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 23:29:41.311567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.311533 6208 generic.go:356] "Generic (PLEG): container finished" podID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerID="7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f" exitCode=0 Nov 03 23:29:41.333262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.333230 6208 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="b28226f4661c7e4c5b3327f04509e38c654a9388ff2a3cae2a2fc865d3a778b3" exitCode=0 Nov 03 23:29:41.333262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.333258 6208 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="3039918425f4a1ef0d62a257e5ea07756f0318a2f3676ffc1f5ad65963ebcc3d" exitCode=0 Nov 03 23:29:41.333262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.333268 6208 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="03aedc760b9401865622845edc19c6cc0ab056306a9dff957d25132187594252" exitCode=0 Nov 03 23:29:41.333262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.333273 6208 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="b7cbd2508f9fa90161effcc922c0a88cc843d0af48c350302b22c9e3a4ff8a6e" exitCode=0 Nov 03 23:29:41.333560 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.333279 6208 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="eed9ffafda1c82c8d3fa69d27e5a96518419c1a78600a06b9fc0a1a353434124" exitCode=0 Nov 03 23:29:41.333560 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.333284 6208 generic.go:356] "Generic (PLEG): container finished" podID="271a3854-3368-425b-9fcd-690738740bb8" containerID="ff90458e67899d62c6943f827cd60559b3946c5dad44e63faaa621bf7020d49b" exitCode=0 Nov 03 23:29:41.337043 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.337003 6208 generic.go:356] "Generic (PLEG): container finished" podID="fdc30eef-1cd6-449c-9aef-d15732f1a2f8" containerID="13ae612c564211ab1e87732dfcb4ab392ef6a5a46a3f8533a560d93c03784d17" exitCode=0 Nov 03 23:29:41.345980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.345949 6208 generic.go:356] "Generic (PLEG): container finished" podID="00d0df6b1c0ad92c712ad5146e3e385d" containerID="c65a1184f38ee2ce849f8f38c3d5380241963f9395c6af9511bb6cb068a8b9c4" exitCode=0 Nov 03 23:29:41.368879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.368805 6208 manager.go:324] Recovery completed Nov 03 23:29:41.386589 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:29:41.386552 6208 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 23:29:41.393312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393287 6208 cpu_manager.go:226] "Starting" policy="none" Nov 03 23:29:41.393312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393308 6208 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 23:29:41.393468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393332 6208 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 23:29:41.393468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393460 6208 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 23:29:41.393574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393470 6208 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 23:29:41.393574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393489 6208 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 23:29:41.393574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393494 6208 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 23:29:41.393574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393500 6208 policy_none.go:50] "Start" Nov 03 23:29:41.393574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393507 6208 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 23:29:41.393574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393516 6208 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 23:29:41.393734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393641 6208 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 23:29:41.393734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393648 6208 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 23:29:41.393734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.393653 6208 policy_none.go:44] "Start" Nov 03 23:29:41.398535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.398501 6208 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 23:29:41.402019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.402004 6208 manager.go:341] "Starting Device Plugin manager" Nov 03 23:29:41.402646 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:29:41.402047 6208 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 23:29:41.402718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.402649 6208 server.go:87] "Starting device plugin registration server" Nov 03 23:29:41.402960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.402950 6208 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 23:29:41.402995 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.402961 6208 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 23:29:41.403076 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.403055 6208 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 23:29:41.403203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.403171 6208 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 23:29:41.403261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.403217 6208 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 23:29:41.403261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.403224 6208 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 23:29:41.403354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.403280 6208 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T23:29:41.40320196Z","UUID":"3d7d9c50-7798-4fae-a0ff-8d309d2ed968","Handler":null,"Name":"","Endpoint":""} Nov 03 23:29:41.405050 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.405005 6208 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 23:29:41.405131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.405063 6208 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 23:29:41.405625 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:29:41.405598 6208 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 23:29:41.503630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.503592 6208 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 23:29:41.505614 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.505594 6208 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 23:29:41.505718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.505628 6208 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 23:29:41.505718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.505640 6208 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-137.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 23:29:41.505718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.505687 6208 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.521433 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.521400 6208 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.521559 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.521491 6208 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.587088 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.587000 6208 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal"] Nov 03 23:29:41.587714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.587667 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" event={"ID":"92a1bdd4dac917849714e10bd3b332b6","Type":"ContainerStarted","Data":"5145f3a64bc6602c4631ae64b6c7924ccbd8d44c072e136d017d1c02b774ca6a"} Nov 03 23:29:41.587867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.587830 6208 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.587867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.587850 6208 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.587867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.587858 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" event={"ID":"00d0df6b1c0ad92c712ad5146e3e385d","Type":"ContainerStarted","Data":"974a5e39153c185fb6dd7030966914640260ecc08ff1d0b207be77ab6c404e69"} Nov 03 23:29:41.588025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.587883 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" event={"ID":"00d0df6b1c0ad92c712ad5146e3e385d","Type":"ContainerDied","Data":"c65a1184f38ee2ce849f8f38c3d5380241963f9395c6af9511bb6cb068a8b9c4"} Nov 03 23:29:41.588025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.587902 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" event={"ID":"00d0df6b1c0ad92c712ad5146e3e385d","Type":"ContainerStarted","Data":"7222e28a9cd1391b3af0c5894d3f7be951ab2be85191b93cf47bda6e3bfa03e5"} Nov 03 23:29:41.610301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.610264 6208 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 23:29:41.610442 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:29:41.610326 6208 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.611802 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.611785 6208 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 23:29:41.611875 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:29:41.611831 6208 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.673241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.673145 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.673241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.673184 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.673241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.673201 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92a1bdd4dac917849714e10bd3b332b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"92a1bdd4dac917849714e10bd3b332b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.773774 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.773725 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.773774 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.773777 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.773985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.773801 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92a1bdd4dac917849714e10bd3b332b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"92a1bdd4dac917849714e10bd3b332b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.773985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.773836 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.773985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.773868 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00d0df6b1c0ad92c712ad5146e3e385d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"00d0df6b1c0ad92c712ad5146e3e385d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:41.773985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:41.773841 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92a1bdd4dac917849714e10bd3b332b6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal\" (UID: \"92a1bdd4dac917849714e10bd3b332b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:42.241699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.241666 6208 apiserver.go:52] "Watching apiserver" Nov 03 23:29:42.260755 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.260721 6208 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 23:29:42.261461 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.261436 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-chg92","openshift-monitoring/node-exporter-dsslm","openshift-network-operator/iptables-alerter-msjvc","openshift-network-diagnostics/network-check-target-s7hm2","openshift-console/console-6c96f49478-btlbq","openshift-ingress-canary/ingress-canary-pqtnl","openshift-monitoring/prometheus-k8s-0","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r","openshift-multus/multus-z2zls","openshift-insights/insights-runtime-extractor-6qpvg","openshift-ovn-kubernetes/ovnkube-node-86q5p","openshift-multus/multus-additional-cni-plugins-nsk4l","openshift-console/console-854967fbd5-4lmhp","openshift-dns/node-resolver-vsrwg","openshift-console/console-7b687d5469-mzgl6","openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl","openshift-monitoring/telemeter-client-5df744b67-frfj2","kube-system/global-pull-secret-syncer-p7zhg","kube-system/konnectivity-agent-6ch5g","kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-5ss7r","openshift-image-registry/node-ca-dbjll","openshift-dns/dns-default-h4spw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal"] Nov 03 23:29:42.270293 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.270268 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fnhvl\"" Nov 03 23:29:42.270498 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.270301 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 23:29:42.270611 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.270551 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w5s9r\"" Nov 03 23:29:42.270771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.270748 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 23:29:42.271738 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.271720 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 23:29:42.272891 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.272852 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 23:29:42.272985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.272913 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 23:29:42.272985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.272958 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6xwn6\"" Nov 03 23:29:42.273610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.273591 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hkcvs\"" Nov 03 23:29:42.273707 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.273679 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b74w2\"" Nov 03 23:29:42.273977 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.273957 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dd24d\"" Nov 03 23:29:42.273977 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.273971 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jctss\"" Nov 03 23:29:42.274743 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.274722 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.274820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.274752 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q987z\"" Nov 03 23:29:42.274820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.274755 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgkz4\"" Nov 03 23:29:42.274909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.274860 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnnt6\"" Nov 03 23:29:42.275384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.275368 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.275456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.275396 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j9hzk\"" Nov 03 23:29:42.275456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.275444 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.275518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.275460 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lsnsx\"" Nov 03 23:29:42.275636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.275621 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.275695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.275653 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.275695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.275679 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 23:29:42.275770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.275694 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 23:29:42.276370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276357 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 23:29:42.276426 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276410 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 23:29:42.276426 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276418 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.276529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276431 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.276529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276450 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.276529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276461 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 23:29:42.276529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276486 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.276529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276493 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.276529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276494 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 23:29:42.276529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276494 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.276529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.276502 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 23:29:42.277111 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.277096 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 23:29:42.277195 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.277181 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.278172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278160 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 23:29:42.278242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278233 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.278307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278242 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 23:29:42.278307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278243 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 23:29:42.278307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278262 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.278307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278261 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.278307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278295 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k4d88\"" Nov 03 23:29:42.278504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278315 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.278504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278328 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.278504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278473 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.278717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278700 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 23:29:42.278770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.278730 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 23:29:42.279817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.279796 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 23:29:42.279888 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.279801 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 23:29:42.279888 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.279801 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 23:29:42.279888 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.279829 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 23:29:42.280003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.279970 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 23:29:42.280003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.279993 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hpfml\"" Nov 03 23:29:42.281445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281395 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x2bch\"" Nov 03 23:29:42.281541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281420 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 23:29:42.281541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281454 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 23:29:42.281541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281454 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 23:29:42.281541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281488 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 23:29:42.281541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281515 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k24zf\"" Nov 03 23:29:42.281775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281762 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.281816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281804 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 23:29:42.281863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281809 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 23:29:42.281863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281849 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 23:29:42.281952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281868 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.281952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281870 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.281952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281885 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9cdhc\"" Nov 03 23:29:42.281952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281885 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 23:29:42.281952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281917 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 23:29:42.281952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.281924 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 23:29:42.282198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.282074 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fw76h\"" Nov 03 23:29:42.283021 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.283006 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 23:29:42.283241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.283224 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.283241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.283235 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 23:29:42.283334 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.283223 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 23:29:42.283334 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.283283 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 23:29:42.283698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.283685 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z6dff\"" Nov 03 23:29:42.285493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.285475 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 23:29:42.285554 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.285502 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 23:29:42.287318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.287303 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 23:29:42.302170 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.302142 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 23:29:42.303065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.303026 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 23:29:42.303862 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.303837 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 23:29:42.303988 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.303967 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 23:29:42.349466 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.349439 6208 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:42.349630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.349553 6208 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:42.356674 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.356656 6208 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 23:29:42.362258 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.362189 6208 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 23:29:42.362258 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:29:42.362238 6208 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:42.363174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.363159 6208 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 23:29:42.363254 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:29:42.363200 6208 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-137.us-east-2.compute.internal" Nov 03 23:29:42.367174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367153 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 23:29:42.367174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367159 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 23:29:42.367723 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367702 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 23:29:42.367804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367740 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 23:29:42.367804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367763 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 23:29:42.367804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367766 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 23:29:42.367804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367786 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 23:29:42.367980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367835 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 23:29:42.367980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367871 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 23:29:42.367980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367911 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-482vvasamrjf7\"" Nov 03 23:29:42.367980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.367974 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cfq6h\"" Nov 03 23:29:42.368110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.368009 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 23:29:42.368110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.368098 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 23:29:42.376232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376213 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysconfig\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.376308 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376240 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fc8829d-b25c-4952-8cd8-816aa35089ac-crio-socket\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.376308 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376262 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.376308 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376281 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp7xk\" (UniqueName: \"kubernetes.io/projected/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-kube-api-access-kp7xk\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:42.376419 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376330 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.376419 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376359 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.376419 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376379 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.376419 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376413 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-conf\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.376593 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376429 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-netns\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.376593 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376447 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-ovn\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.376593 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376517 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.376593 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376577 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44cj\" (UniqueName: \"kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.376784 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376604 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-multus-certs\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.376784 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376625 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.376784 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376626 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.376784 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376654 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkst\" (UniqueName: \"kubernetes.io/projected/36489906-6e8f-4aa6-ae96-f872732808ed-kube-api-access-7kkst\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.376784 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376655 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.376784 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376740 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.377015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376828 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.377015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376838 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.377015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376852 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-kubernetes\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.377015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376884 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.377015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376914 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-socket-dir-parent\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.377015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376929 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-etc-kubernetes\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.377015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.376980 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-socket-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.377263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377022 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.377263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377028 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.377263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377078 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.377263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377175 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-wtmp\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.377263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377204 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.377263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377216 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.377477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377308 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-hostroot\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.377477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377316 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.377477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377345 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-serving-certs-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.377477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377374 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.377477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377462 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.377711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377499 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.377711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377516 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.377711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377549 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-systemd-units\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.377711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377553 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-serving-certs-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.377711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377614 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.377711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377633 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.378012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377747 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.378012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377809 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.378012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377825 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.378012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377843 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt84d\" (UniqueName: \"kubernetes.io/projected/2bc833b9-89ed-4f1a-8dec-602fffd9c072-kube-api-access-pt84d\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:42.378012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377867 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.378012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377937 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cz7\" (UniqueName: \"kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.378012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.377985 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.378263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378063 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-k8s-cni-cncf-io\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.378263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378160 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-multus-daemon-config\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.378263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378177 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-lib-modules\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.378263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378240 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-systemd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.378460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378269 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-netd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.378460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378309 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.378460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378328 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfj2\" (UniqueName: \"kubernetes.io/projected/f42ad3cc-de80-43bb-b548-dfa6c78618e5-kube-api-access-sdfj2\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:42.378460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378348 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-host\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.378460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378383 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fc8829d-b25c-4952-8cd8-816aa35089ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.378460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378396 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-multus-daemon-config\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.378460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378430 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378463 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378483 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-sys\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378517 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wfwh\" (UniqueName: \"kubernetes.io/projected/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-kube-api-access-8wfwh\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378606 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378647 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378682 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wtn\" (UniqueName: \"kubernetes.io/projected/dc219915-7ac3-4e9b-863f-4c641a8b6540-kube-api-access-j8wtn\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378703 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378726 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-var-lib-kubelet\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378735 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378751 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:42.378775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378779 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1855ee11-b7e8-49f7-bf9d-4439ac060f20-hosts-file\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378805 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378831 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.378914 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379092 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379108 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379149 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-registration-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379179 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379206 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379210 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-kubelet\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.379270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379226 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f42ad3cc-de80-43bb-b548-dfa6c78618e5-metrics-certs\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379307 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379331 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379386 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-env-overrides\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379412 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-run\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379412 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9fc8829d-b25c-4952-8cd8-816aa35089ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379443 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379454 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379479 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-metrics-client-ca\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379502 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379547 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzcx\" (UniqueName: \"kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379574 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-cni-binary-copy\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.379699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379640 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-env-overrides\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379733 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379749 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36489906-6e8f-4aa6-ae96-f872732808ed-cni-binary-copy\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379803 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc833b9-89ed-4f1a-8dec-602fffd9c072-tmp-dir\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379845 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379863 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379878 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24tz\" (UniqueName: \"kubernetes.io/projected/34cd9f7a-0794-465b-9443-d6723c5505fb-kube-api-access-m24tz\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379909 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-slash\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379924 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-dbus\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379945 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-cnibin\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379960 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc833b9-89ed-4f1a-8dec-602fffd9c072-tmp-dir\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.379966 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctzq\" (UniqueName: \"kubernetes.io/projected/271a3854-3368-425b-9fcd-690738740bb8-kube-api-access-cctzq\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380108 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380170 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380189 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380213 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e545f4e-ce4d-4a60-ba33-74207fb40c48-konnectivity-ca\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380238 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-textfile\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380264 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.380289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380291 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c46g\" (UniqueName: \"kubernetes.io/projected/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-kube-api-access-5c46g\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380305 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380356 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380359 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-textfile\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380383 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-var-lib-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380415 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-kubelet-config\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380461 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-kubelet\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380496 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380507 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380524 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-metrics-client-ca\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380527 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e545f4e-ce4d-4a60-ba33-74207fb40c48-konnectivity-ca\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380543 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnbk\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380573 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-config\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380600 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-system-cni-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380641 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380678 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380696 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380706 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380710 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc833b9-89ed-4f1a-8dec-602fffd9c072-metrics-tls\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380796 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380840 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380868 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380896 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380907 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc833b9-89ed-4f1a-8dec-602fffd9c072-metrics-tls\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380914 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-log-socket\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380937 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-script-lib\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.380995 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381023 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc219915-7ac3-4e9b-863f-4c641a8b6540-host-slash\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381061 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-node-log\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381088 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-metrics-client-ca\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381092 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381113 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ht52\" (UniqueName: \"kubernetes.io/projected/17b0bd49-65e6-4393-88ba-b578c5eab313-kube-api-access-9ht52\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:42.381140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381147 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381138 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381209 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381237 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-root\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381261 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381287 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5tv\" (UniqueName: \"kubernetes.io/projected/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kube-api-access-rj5tv\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381313 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381336 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-multus\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381362 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381387 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-etc-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381382 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381414 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381438 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381464 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381494 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2zm\" (UniqueName: \"kubernetes.io/projected/3a95c441-120e-44ce-8b68-516030f94a91-kube-api-access-9h2zm\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381520 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-tmp\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381558 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-serviceca\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381690 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381757 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381811 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-tmp\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381879 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a95c441-120e-44ce-8b68-516030f94a91-metrics-client-ca\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381968 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c784845-c509-4b51-86b8-ce9075600208-original-pull-secret\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381969 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.381995 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-bin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382099 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382130 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382150 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382166 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-federate-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382184 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-device-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382226 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382228 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-secret-telemeter-client\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382241 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc219915-7ac3-4e9b-863f-4c641a8b6540-iptables-alerter-script\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382263 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-sys\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382278 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382293 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-bin\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382309 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382327 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382386 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382405 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-cnibin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382415 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-script-lib\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382422 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-systemd\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382438 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1855ee11-b7e8-49f7-bf9d-4439ac060f20-tmp-dir\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382484 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1855ee11-b7e8-49f7-bf9d-4439ac060f20-tmp-dir\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382622 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-telemeter-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382660 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382688 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bj29\" (UniqueName: \"kubernetes.io/projected/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-api-access-9bj29\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382708 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382732 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc833b9-89ed-4f1a-8dec-602fffd9c072-config-volume\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:42.382724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382756 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e545f4e-ce4d-4a60-ba33-74207fb40c48-agent-certs\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382780 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc8829d-b25c-4952-8cd8-816aa35089ac-data-volume\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382795 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382805 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gbd\" (UniqueName: \"kubernetes.io/projected/fecf1197-13f8-4c4a-a878-c784e18385d6-kube-api-access-v4gbd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382835 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-netns\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383058 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/271a3854-3368-425b-9fcd-690738740bb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383090 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-host\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383126 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-system-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383157 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-modprobe-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383185 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b0bd49-65e6-4393-88ba-b578c5eab313-cert\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383213 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fecf1197-13f8-4c4a-a878-c784e18385d6-ovn-node-metrics-cert\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383245 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383277 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqglk\" (UniqueName: \"kubernetes.io/projected/1855ee11-b7e8-49f7-bf9d-4439ac060f20-kube-api-access-gqglk\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383307 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383324 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383405 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-tls\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383435 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383454 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-sys-fs\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383470 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-tuned\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383486 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383503 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383100 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e545f4e-ce4d-4a60-ba33-74207fb40c48-agent-certs\") pod \"konnectivity-agent-6ch5g\" (UID: \"1e545f4e-ce4d-4a60-ba33-74207fb40c48\") " pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383527 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.382886 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383545 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383562 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-conf-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383566 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fecf1197-13f8-4c4a-a878-c784e18385d6-ovnkube-config\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383628 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-tuned\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383786 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-serviceca\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383809 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc833b9-89ed-4f1a-8dec-602fffd9c072-config-volume\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383858 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.383977 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-tls\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384065 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384131 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384187 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384303 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384323 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc219915-7ac3-4e9b-863f-4c641a8b6540-iptables-alerter-script\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384443 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384577 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3a95c441-120e-44ce-8b68-516030f94a91-federate-client-tls\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384598 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384644 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fc8829d-b25c-4952-8cd8-816aa35089ac-data-volume\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384685 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384852 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384864 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384866 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.384939 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fecf1197-13f8-4c4a-a878-c784e18385d6-ovn-node-metrics-cert\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.384994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.385004 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b0bd49-65e6-4393-88ba-b578c5eab313-cert\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:42.391703 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.391684 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp7xk\" (UniqueName: \"kubernetes.io/projected/37f77e6f-4bd8-4a9e-ab23-c1337365cac1-kube-api-access-kp7xk\") pod \"prometheus-operator-9cd89b74c-fzfcl\" (UID: \"37f77e6f-4bd8-4a9e-ab23-c1337365cac1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fzfcl" Nov 03 23:29:42.404807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.404782 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfj2\" (UniqueName: \"kubernetes.io/projected/f42ad3cc-de80-43bb-b548-dfa6c78618e5-kube-api-access-sdfj2\") pod \"network-metrics-daemon-chg92\" (UID: \"f42ad3cc-de80-43bb-b548-dfa6c78618e5\") " pod="openshift-multus/network-metrics-daemon-chg92" Nov 03 23:29:42.413100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.412293 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wtn\" (UniqueName: \"kubernetes.io/projected/dc219915-7ac3-4e9b-863f-4c641a8b6540-kube-api-access-j8wtn\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:29:42.413100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.412363 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctzq\" (UniqueName: \"kubernetes.io/projected/271a3854-3368-425b-9fcd-690738740bb8-kube-api-access-cctzq\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.413100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.412667 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt84d\" (UniqueName: \"kubernetes.io/projected/2bc833b9-89ed-4f1a-8dec-602fffd9c072-kube-api-access-pt84d\") pod \"dns-default-h4spw\" (UID: \"2bc833b9-89ed-4f1a-8dec-602fffd9c072\") " pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:42.413100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.412762 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzcx\" (UniqueName: \"kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx\") pod \"console-6c96f49478-btlbq\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:29:42.413100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.412829 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c46g\" (UniqueName: \"kubernetes.io/projected/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-kube-api-access-5c46g\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:29:42.413100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.413004 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6f7\" (UniqueName: \"kubernetes.io/projected/72d93d4f-2af9-44d5-b8e5-542b30760f34-kube-api-access-kr6f7\") pod \"network-check-target-s7hm2\" (UID: \"72d93d4f-2af9-44d5-b8e5-542b30760f34\") " pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:42.413100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.413062 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44cj\" (UniqueName: \"kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj\") pod \"console-854967fbd5-4lmhp\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:42.413507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.413199 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cz7\" (UniqueName: \"kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7\") pod \"console-7b687d5469-mzgl6\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:42.413507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.413238 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wfwh\" (UniqueName: \"kubernetes.io/projected/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-kube-api-access-8wfwh\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.413507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.413245 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkst\" (UniqueName: \"kubernetes.io/projected/36489906-6e8f-4aa6-ae96-f872732808ed-kube-api-access-7kkst\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.418992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.418517 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24tz\" (UniqueName: \"kubernetes.io/projected/34cd9f7a-0794-465b-9443-d6723c5505fb-kube-api-access-m24tz\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.418992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.418526 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj5tv\" (UniqueName: \"kubernetes.io/projected/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kube-api-access-rj5tv\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.418992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.418697 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gbd\" (UniqueName: \"kubernetes.io/projected/fecf1197-13f8-4c4a-a878-c784e18385d6-kube-api-access-v4gbd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.418992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.418951 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnbk\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk\") pod \"prometheus-k8s-0\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:42.420858 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.420839 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2zm\" (UniqueName: \"kubernetes.io/projected/3a95c441-120e-44ce-8b68-516030f94a91-kube-api-access-9h2zm\") pod \"telemeter-client-5df744b67-frfj2\" (UID: \"3a95c441-120e-44ce-8b68-516030f94a91\") " pod="openshift-monitoring/telemeter-client-5df744b67-frfj2" Nov 03 23:29:42.421745 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.421723 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqglk\" (UniqueName: \"kubernetes.io/projected/1855ee11-b7e8-49f7-bf9d-4439ac060f20-kube-api-access-gqglk\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:29:42.422025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.422007 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ht52\" (UniqueName: \"kubernetes.io/projected/17b0bd49-65e6-4393-88ba-b578c5eab313-kube-api-access-9ht52\") pod \"ingress-canary-pqtnl\" (UID: \"17b0bd49-65e6-4393-88ba-b578c5eab313\") " pod="openshift-ingress-canary/ingress-canary-pqtnl" Nov 03 23:29:42.429239 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.429218 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bj29\" (UniqueName: \"kubernetes.io/projected/9fc8829d-b25c-4952-8cd8-816aa35089ac-kube-api-access-9bj29\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.483960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.483925 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-lib-modules\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.483960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.483968 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-systemd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.483985 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-netd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484003 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-host\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484022 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484060 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484082 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-sys\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484105 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-var-lib-kubelet\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484127 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1855ee11-b7e8-49f7-bf9d-4439ac060f20-hosts-file\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484137 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-lib-modules\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484151 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484199 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-registration-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484214 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484232 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-kubelet\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484249 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484253 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-systemd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484265 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-run\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484279 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484286 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-netd\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484307 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484322 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-slash\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484326 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-host\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484337 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-dbus\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484355 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-cnibin\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484362 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484380 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-var-lib-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484399 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-kubelet-config\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484407 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484421 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-kubelet\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484437 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-var-lib-kubelet\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484446 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484450 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-system-cni-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484485 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-sys\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484466 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-run\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484509 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-cnibin\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484520 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-kubelet-config\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484520 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-log-socket\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484461 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1855ee11-b7e8-49f7-bf9d-4439ac060f20-hosts-file\") pod \"node-resolver-vsrwg\" (UID: \"1855ee11-b7e8-49f7-bf9d-4439ac060f20\") " pod="openshift-dns/node-resolver-vsrwg" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484525 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c784845-c509-4b51-86b8-ce9075600208-dbus\") pod \"global-pull-secret-syncer-p7zhg\" (UID: \"9c784845-c509-4b51-86b8-ce9075600208\") " pod="kube-system/global-pull-secret-syncer-p7zhg" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484556 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-kubelet\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484563 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-slash\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484557 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/271a3854-3368-425b-9fcd-690738740bb8-system-cni-dir\") pod \"multus-additional-cni-plugins-nsk4l\" (UID: \"271a3854-3368-425b-9fcd-690738740bb8\") " pod="openshift-multus/multus-additional-cni-plugins-nsk4l" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484589 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc219915-7ac3-4e9b-863f-4c641a8b6540-host-slash\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484594 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-kubelet\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484600 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484605 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-var-lib-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484592 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-registration-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484640 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-node-log\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484647 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc219915-7ac3-4e9b-863f-4c641a8b6540-host-slash\") pod \"iptables-alerter-msjvc\" (UID: \"dc219915-7ac3-4e9b-863f-4c641a8b6540\") " pod="openshift-network-operator/iptables-alerter-msjvc" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484642 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484670 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-log-socket\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484718 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-node-log\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484743 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-root\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484765 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-root\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484785 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-multus\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484812 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-multus\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.484797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484816 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-etc-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484834 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-etc-openvswitch\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484842 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484859 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-bin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484872 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484889 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-device-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484896 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-var-lib-cni-bin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484906 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-sys\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484920 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-bin\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484925 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-device-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484947 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-cni-bin\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484955 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-sys\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.484977 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-cnibin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485011 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-systemd\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485060 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-netns\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485068 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-cnibin\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485074 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-systemd\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485084 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-host\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485104 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-netns\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485106 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-system-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485140 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-modprobe-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485161 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a707ad-9256-4eb1-bf0f-eae7ab6c8d32-host\") pod \"node-ca-dbjll\" (UID: \"58a707ad-9256-4eb1-bf0f-eae7ab6c8d32\") " pod="openshift-image-registry/node-ca-dbjll" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485179 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-sys-fs\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485186 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-system-cni-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485195 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-conf-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485210 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysconfig\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485222 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-sys-fs\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485224 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fc8829d-b25c-4952-8cd8-816aa35089ac-crio-socket\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485253 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9fc8829d-b25c-4952-8cd8-816aa35089ac-crio-socket\") pod \"insights-runtime-extractor-6qpvg\" (UID: \"9fc8829d-b25c-4952-8cd8-816aa35089ac\") " pod="openshift-insights/insights-runtime-extractor-6qpvg" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485254 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-modprobe-d\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485253 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-conf-dir\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485269 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysconfig\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485269 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-conf\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485307 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-netns\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485331 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-ovn\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485341 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-host-run-netns\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485350 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-sysctl-conf\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485355 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-multus-certs\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485366 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-run-ovn\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485381 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-kubernetes\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485403 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-multus-certs\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485409 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-socket-dir-parent\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485444 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34cd9f7a-0794-465b-9443-d6723c5505fb-etc-kubernetes\") pod \"tuned-5ss7r\" (UID: \"34cd9f7a-0794-465b-9443-d6723c5505fb\") " pod="openshift-cluster-node-tuning-operator/tuned-5ss7r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485467 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-multus-socket-dir-parent\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485479 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-etc-kubernetes\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485495 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-socket-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485509 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-wtmp\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485509 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-etc-kubernetes\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485542 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-hostroot\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485552 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fdc30eef-1cd6-449c-9aef-d15732f1a2f8-node-exporter-wtmp\") pod \"node-exporter-dsslm\" (UID: \"fdc30eef-1cd6-449c-9aef-d15732f1a2f8\") " pod="openshift-monitoring/node-exporter-dsslm" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485553 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-socket-dir\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485560 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485584 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-hostroot\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485586 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ceed0dfc-9a74-469f-a91d-aab8c3fa402d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs64r\" (UID: \"ceed0dfc-9a74-469f-a91d-aab8c3fa402d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs64r" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485597 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-systemd-units\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485618 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fecf1197-13f8-4c4a-a878-c784e18385d6-systemd-units\") pod \"ovnkube-node-86q5p\" (UID: \"fecf1197-13f8-4c4a-a878-c784e18385d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485626 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-k8s-cni-cncf-io\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.486356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.485686 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/36489906-6e8f-4aa6-ae96-f872732808ed-host-run-k8s-cni-cncf-io\") pod \"multus-z2zls\" (UID: \"36489906-6e8f-4aa6-ae96-f872732808ed\") " pod="openshift-multus/multus-z2zls" Nov 03 23:29:42.593065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.592968 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b687d5469-mzgl6"] Nov 03 23:29:42.912786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:42.912705 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:43.565962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:43.565933 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h4spw" Nov 03 23:29:44.280164 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:44.280134 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:44.283796 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:44.283759 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:44.296072 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:44.296051 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:29:44.441451 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:44.441406 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:44.647062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:44.646965 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:44.787624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:44.787574 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:44.895931 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:44.895899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:45.002051 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:45.001998 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:47.785018 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:47.784981 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:47.785376 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:47.785260 6208 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 23:29:48.617350 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:48.617313 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:48.865399 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:48.865361 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:48.971771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:48.971728 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:50.032255 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:50.032166 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:29:51.925204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:51.925156 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-86q5p" Nov 03 23:29:52.266398 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:52.266371 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:29:52.317387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:52.317355 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:55.646742 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:55.646700 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:55.745369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:55.745336 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:57.071158 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.071124 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86b9b49bc8-j9wrj"] Nov 03 23:29:57.076339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.076321 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.091772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.091747 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86b9b49bc8-j9wrj"] Nov 03 23:29:57.185340 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.185298 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:57.224398 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.224358 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-oauth-serving-cert\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.224567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.224413 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2378e029-438f-4f14-93b2-5b65a289eb40-console-serving-cert\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.224567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.224432 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2378e029-438f-4f14-93b2-5b65a289eb40-console-oauth-config\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.224567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.224480 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnh84\" (UniqueName: \"kubernetes.io/projected/2378e029-438f-4f14-93b2-5b65a289eb40-kube-api-access-rnh84\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.224567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.224544 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-service-ca\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.224718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.224609 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-console-config\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.224718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.224649 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-trusted-ca-bundle\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.224718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.224692 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2378e029-438f-4f14-93b2-5b65a289eb40-tmp\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.297591 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.297550 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:57.325203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325110 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-oauth-serving-cert\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.325203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325163 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2378e029-438f-4f14-93b2-5b65a289eb40-console-serving-cert\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.325203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325186 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2378e029-438f-4f14-93b2-5b65a289eb40-console-oauth-config\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.325203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325203 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnh84\" (UniqueName: \"kubernetes.io/projected/2378e029-438f-4f14-93b2-5b65a289eb40-kube-api-access-rnh84\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.325544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325231 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-service-ca\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.325544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325302 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-console-config\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.325544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325338 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-trusted-ca-bundle\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.325544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325375 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2378e029-438f-4f14-93b2-5b65a289eb40-tmp\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.325744 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325655 6208 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 23:29:57.325872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.325849 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2378e029-438f-4f14-93b2-5b65a289eb40-tmp\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.326100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.326076 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-service-ca\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.326211 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.326120 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-oauth-serving-cert\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.326378 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.326359 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-console-config\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.326522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.326503 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2378e029-438f-4f14-93b2-5b65a289eb40-trusted-ca-bundle\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.327868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.327845 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2378e029-438f-4f14-93b2-5b65a289eb40-console-oauth-config\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.327954 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.327929 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2378e029-438f-4f14-93b2-5b65a289eb40-console-serving-cert\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.338354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.338334 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnh84\" (UniqueName: \"kubernetes.io/projected/2378e029-438f-4f14-93b2-5b65a289eb40-kube-api-access-rnh84\") pod \"console-86b9b49bc8-j9wrj\" (UID: \"2378e029-438f-4f14-93b2-5b65a289eb40\") " pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.386844 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.386802 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:29:57.658290 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.658215 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86b9b49bc8-j9wrj"] Nov 03 23:29:57.661589 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:29:57.661563 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2378e029_438f_4f14_93b2_5b65a289eb40.slice/crio-048b25e48e51e0c05cb02b407b8ea9aa344da6c446ce746cd558131b36cd5f7f WatchSource:0}: Error finding container 048b25e48e51e0c05cb02b407b8ea9aa344da6c446ce746cd558131b36cd5f7f: Status 404 returned error can't find the container with id 048b25e48e51e0c05cb02b407b8ea9aa344da6c446ce746cd558131b36cd5f7f Nov 03 23:29:57.803083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:57.803046 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:58.411232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:58.411198 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b9b49bc8-j9wrj" event={"ID":"2378e029-438f-4f14-93b2-5b65a289eb40","Type":"ContainerStarted","Data":"74b738183f017833b1cecdbe8efbd7b4245989d7889a00b955390c2dc87c3d50"} Nov 03 23:29:58.411624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:58.411244 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b9b49bc8-j9wrj" event={"ID":"2378e029-438f-4f14-93b2-5b65a289eb40","Type":"ContainerStarted","Data":"048b25e48e51e0c05cb02b407b8ea9aa344da6c446ce746cd558131b36cd5f7f"} Nov 03 23:29:58.432680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:58.432618 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86b9b49bc8-j9wrj" podStartSLOduration=1.432604857 podStartE2EDuration="1.432604857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:29:58.432293508 +0000 UTC m=+17.254889671" watchObservedRunningTime="2026-11-03 23:29:58.432604857 +0000 UTC m=+17.255201090" Nov 03 23:29:58.621997 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:58.621965 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:58.625732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:58.625708 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:29:58.780457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:58.780420 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:58.898366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:58.898328 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:59.623648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:59.623600 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:59.727242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:59.727205 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:59.886211 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:59.886125 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:59.925617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:29:59.925581 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:00.003913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:00.003879 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:00.596281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:00.596246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:00.708583 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:00.708544 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:04.376228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:04.376188 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 23:30:04.376641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:04.376582 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="prometheus" containerID="cri-o://9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5" gracePeriod=600 Nov 03 23:30:04.376714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:04.376609 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="thanos-sidecar" containerID="cri-o://5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e" gracePeriod=600 Nov 03 23:30:04.376714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:04.376658 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="config-reloader" containerID="cri-o://2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457" gracePeriod=600 Nov 03 23:30:04.376714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:04.376649 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a" gracePeriod=600 Nov 03 23:30:04.376714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:04.376609 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy" containerID="cri-o://cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58" gracePeriod=600 Nov 03 23:30:04.376914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:04.376656 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy-web" containerID="cri-o://966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20" gracePeriod=600 Nov 03 23:30:05.440150 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:05.440119 6208 generic.go:356] "Generic (PLEG): container finished" podID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerID="966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20" exitCode=0 Nov 03 23:30:05.440150 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:05.440146 6208 generic.go:356] "Generic (PLEG): container finished" podID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerID="5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e" exitCode=0 Nov 03 23:30:05.440150 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:05.440152 6208 generic.go:356] "Generic (PLEG): container finished" podID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerID="2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457" exitCode=0 Nov 03 23:30:05.440657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:05.440196 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerDied","Data":"966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20"} Nov 03 23:30:05.440657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:05.440278 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerDied","Data":"5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e"} Nov 03 23:30:05.440657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:05.440329 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerDied","Data":"2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457"} Nov 03 23:30:05.703399 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:05.703308 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:30:06.446472 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:06.446438 6208 generic.go:356] "Generic (PLEG): container finished" podID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerID="d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a" exitCode=0 Nov 03 23:30:06.446472 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:06.446462 6208 generic.go:356] "Generic (PLEG): container finished" podID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerID="cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58" exitCode=0 Nov 03 23:30:06.446873 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:06.446507 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerDied","Data":"d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a"} Nov 03 23:30:06.446873 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:06.446570 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerDied","Data":"cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58"} Nov 03 23:30:07.162627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.162595 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:07.259194 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.259156 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:07.293239 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.293201 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c96f49478-btlbq" podUID="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" containerName="console" containerID="cri-o://c2d5eadc419d726416fde7d70cfc7cb9cd8f80edc150149569fe098c5ef95443" gracePeriod=15 Nov 03 23:30:07.386922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.386889 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:30:07.387085 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.386949 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:30:07.391887 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.391862 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:30:07.451835 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.451798 6208 generic.go:356] "Generic (PLEG): container finished" podID="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" containerID="c2d5eadc419d726416fde7d70cfc7cb9cd8f80edc150149569fe098c5ef95443" exitCode=0 Nov 03 23:30:07.452300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.451873 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c96f49478-btlbq" event={"ID":"c38365ee-cbdb-4999-a6c9-e78aa4552bb9","Type":"ContainerDied","Data":"c2d5eadc419d726416fde7d70cfc7cb9cd8f80edc150149569fe098c5ef95443"} Nov 03 23:30:07.455946 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.455925 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86b9b49bc8-j9wrj" Nov 03 23:30:07.527912 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.527882 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854967fbd5-4lmhp"] Nov 03 23:30:07.567224 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.567197 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:30:07.609758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.609727 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config\") pod \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " Nov 03 23:30:07.609922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.609762 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca\") pod \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " Nov 03 23:30:07.609922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.609785 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp\") pod \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " Nov 03 23:30:07.609922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.609833 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config\") pod \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " Nov 03 23:30:07.609922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.609874 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert\") pod \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " Nov 03 23:30:07.610131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.609946 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert\") pod \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " Nov 03 23:30:07.610131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.610001 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle\") pod \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " Nov 03 23:30:07.610131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.610029 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx\" (UniqueName: \"kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx\") pod \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\" (UID: \"c38365ee-cbdb-4999-a6c9-e78aa4552bb9\") " Nov 03 23:30:07.610297 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.610218 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp" pod "c38365ee-cbdb-4999-a6c9-e78aa4552bb9" (UID: "c38365ee-cbdb-4999-a6c9-e78aa4552bb9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:07.610353 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.610296 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca" pod "c38365ee-cbdb-4999-a6c9-e78aa4552bb9" (UID: "c38365ee-cbdb-4999-a6c9-e78aa4552bb9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:07.610403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.610344 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config" pod "c38365ee-cbdb-4999-a6c9-e78aa4552bb9" (UID: "c38365ee-cbdb-4999-a6c9-e78aa4552bb9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:07.610403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.610352 6208 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-tmp\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:07.610403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.610380 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert" pod "c38365ee-cbdb-4999-a6c9-e78aa4552bb9" (UID: "c38365ee-cbdb-4999-a6c9-e78aa4552bb9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:07.610760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.610732 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle" pod "c38365ee-cbdb-4999-a6c9-e78aa4552bb9" (UID: "c38365ee-cbdb-4999-a6c9-e78aa4552bb9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:07.612147 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.612122 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config" pod "c38365ee-cbdb-4999-a6c9-e78aa4552bb9" (UID: "c38365ee-cbdb-4999-a6c9-e78aa4552bb9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:07.612332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.612315 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert" pod "c38365ee-cbdb-4999-a6c9-e78aa4552bb9" (UID: "c38365ee-cbdb-4999-a6c9-e78aa4552bb9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:07.612389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.612352 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx" pod "c38365ee-cbdb-4999-a6c9-e78aa4552bb9" (UID: "c38365ee-cbdb-4999-a6c9-e78aa4552bb9"). InnerVolumeSpecName "kube-api-access-lfzcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:07.711610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.711576 6208 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-serving-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:07.711778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.711640 6208 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-trusted-ca-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:07.711778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.711655 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfzcx\" (UniqueName: \"kubernetes.io/projected/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-kube-api-access-lfzcx\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:07.711778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.711669 6208 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-oauth-config\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:07.711778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.711681 6208 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:07.711778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.711692 6208 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-console-config\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:07.711778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:07.711705 6208 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c38365ee-cbdb-4999-a6c9-e78aa4552bb9-oauth-serving-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:08.371285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.371245 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7b687d5469-mzgl6" podUID="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" containerName="console" containerID="cri-o://928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f" gracePeriod=15 Nov 03 23:30:08.456566 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.456534 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c96f49478-btlbq" event={"ID":"c38365ee-cbdb-4999-a6c9-e78aa4552bb9","Type":"ContainerDied","Data":"92c0ec3dbccb50c66767557717dd84cdb65d12e65edd70a84cd706b09493716a"} Nov 03 23:30:08.456947 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.456567 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c96f49478-btlbq" Nov 03 23:30:08.456947 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.456590 6208 scope.go:122] "RemoveContainer" containerID="c2d5eadc419d726416fde7d70cfc7cb9cd8f80edc150149569fe098c5ef95443" Nov 03 23:30:08.482558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.482532 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c96f49478-btlbq"] Nov 03 23:30:08.486986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.486965 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c96f49478-btlbq"] Nov 03 23:30:08.670777 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.670752 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:30:08.719664 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.719633 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config\") pod \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " Nov 03 23:30:08.719664 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.719669 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7\" (UniqueName: \"kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7\") pod \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " Nov 03 23:30:08.719876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.719707 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca\") pod \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " Nov 03 23:30:08.719876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.719729 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config\") pod \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " Nov 03 23:30:08.719876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.719748 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert\") pod \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " Nov 03 23:30:08.719876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.719793 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert\") pod \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " Nov 03 23:30:08.719876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.719825 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle\") pod \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " Nov 03 23:30:08.720157 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.719875 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp\") pod \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\" (UID: \"d8bfaf2a-e644-40b1-84cd-f2d7d0417207\") " Nov 03 23:30:08.720261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.720235 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca" pod "d8bfaf2a-e644-40b1-84cd-f2d7d0417207" (UID: "d8bfaf2a-e644-40b1-84cd-f2d7d0417207"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:08.720355 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.720329 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config" pod "d8bfaf2a-e644-40b1-84cd-f2d7d0417207" (UID: "d8bfaf2a-e644-40b1-84cd-f2d7d0417207"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:08.720403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.720348 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp" pod "d8bfaf2a-e644-40b1-84cd-f2d7d0417207" (UID: "d8bfaf2a-e644-40b1-84cd-f2d7d0417207"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:08.720403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.720361 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert" pod "d8bfaf2a-e644-40b1-84cd-f2d7d0417207" (UID: "d8bfaf2a-e644-40b1-84cd-f2d7d0417207"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:08.720403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.720372 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle" pod "d8bfaf2a-e644-40b1-84cd-f2d7d0417207" (UID: "d8bfaf2a-e644-40b1-84cd-f2d7d0417207"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:08.721974 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.721950 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7" pod "d8bfaf2a-e644-40b1-84cd-f2d7d0417207" (UID: "d8bfaf2a-e644-40b1-84cd-f2d7d0417207"). InnerVolumeSpecName "kube-api-access-76cz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:08.722223 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.722104 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert" pod "d8bfaf2a-e644-40b1-84cd-f2d7d0417207" (UID: "d8bfaf2a-e644-40b1-84cd-f2d7d0417207"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:08.722223 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.722184 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config" pod "d8bfaf2a-e644-40b1-84cd-f2d7d0417207" (UID: "d8bfaf2a-e644-40b1-84cd-f2d7d0417207"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:08.820883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.820843 6208 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-tmp\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:08.820883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.820879 6208 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-oauth-config\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:08.821122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.820893 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76cz7\" (UniqueName: \"kubernetes.io/projected/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-kube-api-access-76cz7\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:08.821122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.820909 6208 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:08.821122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.820921 6208 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-config\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:08.821122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.820933 6208 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-oauth-serving-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:08.821122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.820943 6208 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-console-serving-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:08.821122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:08.820957 6208 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bfaf2a-e644-40b1-84cd-f2d7d0417207-trusted-ca-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:09.291022 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.290986 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" path="/var/lib/kubelet/pods/c38365ee-cbdb-4999-a6c9-e78aa4552bb9/volumes" Nov 03 23:30:09.461929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.461900 6208 generic.go:356] "Generic (PLEG): container finished" podID="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" containerID="928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f" exitCode=0 Nov 03 23:30:09.462412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.461945 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b687d5469-mzgl6" event={"ID":"d8bfaf2a-e644-40b1-84cd-f2d7d0417207","Type":"ContainerDied","Data":"928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f"} Nov 03 23:30:09.462412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.461984 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b687d5469-mzgl6" event={"ID":"d8bfaf2a-e644-40b1-84cd-f2d7d0417207","Type":"ContainerDied","Data":"bce424346a312c6e62eaa196c1114b1df85e40966486216b215ab9b8b02034af"} Nov 03 23:30:09.462412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.462003 6208 scope.go:122] "RemoveContainer" containerID="928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f" Nov 03 23:30:09.462412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.461948 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b687d5469-mzgl6" Nov 03 23:30:09.469628 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.469609 6208 scope.go:122] "RemoveContainer" containerID="928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f" Nov 03 23:30:09.469890 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:09.469866 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f\": container with ID starting with 928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f not found: ID does not exist" containerID="928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f" Nov 03 23:30:09.469984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.469898 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f"} err="failed to get container status \"928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f\": rpc error: code = NotFound desc = could not find container \"928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f\": container with ID starting with 928ce3984525377349bf3abf14118782f5b27fd85a79f2072fc67201a964c82f not found: ID does not exist" Nov 03 23:30:09.480292 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.480271 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b687d5469-mzgl6"] Nov 03 23:30:09.484345 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:09.484325 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b687d5469-mzgl6"] Nov 03 23:30:10.035712 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:10.035687 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s7hm2" Nov 03 23:30:11.016377 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.016348 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6ch5g" Nov 03 23:30:11.291497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.291422 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" path="/var/lib/kubelet/pods/d8bfaf2a-e644-40b1-84cd-f2d7d0417207/volumes" Nov 03 23:30:11.653656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.653618 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:11.744480 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744438 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744495 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744543 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744578 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744616 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744649 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744681 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744709 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744748 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744781 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744814 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744845 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744875 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744907 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744858 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:11.744941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744939 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.745456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744971 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.745456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.744990 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:11.745456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.745013 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.745456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.745072 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.745456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.745103 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.745456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.745134 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db\") pod \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\" (UID: \"f866b162-3fe8-4a69-9ae0-387e0c18d16d\") " Nov 03 23:30:11.745456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.745390 6208 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.745456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.745408 6208 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.745456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.745441 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:11.746644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.746615 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:11.746769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.746711 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:11.746840 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.746789 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:11.747011 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.746882 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:11.747198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.747172 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:11.749559 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.749534 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:11.749721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.749629 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:11.749844 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.749804 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:11.750171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.750123 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:11.750301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.750275 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:11.750356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.750337 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:11.750498 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.750473 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:11.750600 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.750561 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:11.750646 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.750627 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:11.750812 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.750793 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "kube-api-access-qqnbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:11.750882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.750861 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:11.758818 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.758794 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config" pod "f866b162-3fe8-4a69-9ae0-387e0c18d16d" (UID: "f866b162-3fe8-4a69-9ae0-387e0c18d16d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:11.846444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846365 6208 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-metrics-client-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846397 6208 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846408 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqnbk\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-kube-api-access-qqnbk\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846419 6208 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846429 6208 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f866b162-3fe8-4a69-9ae0-387e0c18d16d-tls-assets\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846438 6208 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-metrics-client-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846448 6208 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846477 6208 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-web-config\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846487 6208 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846498 6208 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846507 6208 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-grpc-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846516 6208 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config-out\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846525 6208 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-kube-rbac-proxy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846533 6208 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-db\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846541 6208 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-config\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846550 6208 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866b162-3fe8-4a69-9ae0-387e0c18d16d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846559 6208 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:11.846769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:11.846568 6208 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866b162-3fe8-4a69-9ae0-387e0c18d16d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:12.477278 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.477246 6208 generic.go:356] "Generic (PLEG): container finished" podID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerID="9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5" exitCode=0 Nov 03 23:30:12.477734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.477324 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerDied","Data":"9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5"} Nov 03 23:30:12.477734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.477338 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.477734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.477392 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f866b162-3fe8-4a69-9ae0-387e0c18d16d","Type":"ContainerDied","Data":"b5f3d49f7567843143cc40a2ee82a15d7349002d5fca297f2e2bd6c81a1d6eca"} Nov 03 23:30:12.477734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.477425 6208 scope.go:122] "RemoveContainer" containerID="d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a" Nov 03 23:30:12.484584 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.484554 6208 scope.go:122] "RemoveContainer" containerID="cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58" Nov 03 23:30:12.496121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.496102 6208 scope.go:122] "RemoveContainer" containerID="966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20" Nov 03 23:30:12.501832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.501812 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 23:30:12.502745 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.502717 6208 scope.go:122] "RemoveContainer" containerID="5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e" Nov 03 23:30:12.507979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.507942 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 23:30:12.510241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.510115 6208 scope.go:122] "RemoveContainer" containerID="2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457" Nov 03 23:30:12.516586 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.516569 6208 scope.go:122] "RemoveContainer" containerID="9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5" Nov 03 23:30:12.523598 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.523576 6208 scope.go:122] "RemoveContainer" containerID="7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f" Nov 03 23:30:12.529648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.529632 6208 scope.go:122] "RemoveContainer" containerID="d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a" Nov 03 23:30:12.529900 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:12.529885 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a\": container with ID starting with d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a not found: ID does not exist" containerID="d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a" Nov 03 23:30:12.529942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.529910 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a"} err="failed to get container status \"d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a\": rpc error: code = NotFound desc = could not find container \"d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a\": container with ID starting with d5a528dbb4b85fddd38fe63ea65cef81f430f40cfcd2c346f217dd744ed4e76a not found: ID does not exist" Nov 03 23:30:12.529942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.529930 6208 scope.go:122] "RemoveContainer" containerID="cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58" Nov 03 23:30:12.530168 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:12.530148 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58\": container with ID starting with cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58 not found: ID does not exist" containerID="cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58" Nov 03 23:30:12.530272 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.530171 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58"} err="failed to get container status \"cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58\": rpc error: code = NotFound desc = could not find container \"cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58\": container with ID starting with cbf9641c0557384c80946114f3069fe86ea241ab2ae7cc4c99c6d76bda439e58 not found: ID does not exist" Nov 03 23:30:12.530272 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.530192 6208 scope.go:122] "RemoveContainer" containerID="966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20" Nov 03 23:30:12.530453 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:12.530435 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20\": container with ID starting with 966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20 not found: ID does not exist" containerID="966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20" Nov 03 23:30:12.530513 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.530462 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20"} err="failed to get container status \"966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20\": rpc error: code = NotFound desc = could not find container \"966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20\": container with ID starting with 966efe5eea3e864316af857f82cc5123ef40da36cb95f40237b80b189156de20 not found: ID does not exist" Nov 03 23:30:12.530513 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.530482 6208 scope.go:122] "RemoveContainer" containerID="5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e" Nov 03 23:30:12.530780 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:12.530759 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e\": container with ID starting with 5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e not found: ID does not exist" containerID="5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e" Nov 03 23:30:12.530860 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.530782 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e"} err="failed to get container status \"5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e\": rpc error: code = NotFound desc = could not find container \"5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e\": container with ID starting with 5749d32e3e77a2d30dd3ec33da2d96ff31b48f911dbe5150885b32147a41ee7e not found: ID does not exist" Nov 03 23:30:12.530860 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.530795 6208 scope.go:122] "RemoveContainer" containerID="2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457" Nov 03 23:30:12.531020 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:12.531004 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457\": container with ID starting with 2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457 not found: ID does not exist" containerID="2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457" Nov 03 23:30:12.531092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.531023 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457"} err="failed to get container status \"2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457\": rpc error: code = NotFound desc = could not find container \"2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457\": container with ID starting with 2ec384d8e5d985ae3348d4ef0203112d007f6fdc5f68a77a0654affba0910457 not found: ID does not exist" Nov 03 23:30:12.531092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.531051 6208 scope.go:122] "RemoveContainer" containerID="9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5" Nov 03 23:30:12.531323 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:12.531309 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5\": container with ID starting with 9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5 not found: ID does not exist" containerID="9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5" Nov 03 23:30:12.531371 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.531325 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5"} err="failed to get container status \"9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5\": rpc error: code = NotFound desc = could not find container \"9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5\": container with ID starting with 9329f39365ae4cd8c417cf6712af9d6ceb503248e1770bab2d44ce37f6d16ee5 not found: ID does not exist" Nov 03 23:30:12.531371 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.531336 6208 scope.go:122] "RemoveContainer" containerID="7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f" Nov 03 23:30:12.531531 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:12.531514 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f\": container with ID starting with 7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f not found: ID does not exist" containerID="7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f" Nov 03 23:30:12.531569 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.531534 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f"} err="failed to get container status \"7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f\": rpc error: code = NotFound desc = could not find container \"7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f\": container with ID starting with 7e46d33c3d855f736ee45934bf581b8aa3877f8f9afea8e3bc3e6da27f3f036f not found: ID does not exist" Nov 03 23:30:12.544957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.544936 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 23:30:12.545227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545213 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="prometheus" Nov 03 23:30:12.545281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545228 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="prometheus" Nov 03 23:30:12.545281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545237 6208 cpu_manager.go:414] "containerMap: removing container" podUID="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" containerName="console" Nov 03 23:30:12.545281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545242 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" containerName="console" Nov 03 23:30:12.545281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545255 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy" Nov 03 23:30:12.545281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545260 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy" Nov 03 23:30:12.545281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545269 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="thanos-sidecar" Nov 03 23:30:12.545281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545277 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="thanos-sidecar" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545287 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy-web" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545292 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy-web" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545300 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="init-config-reloader" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545304 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="init-config-reloader" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545313 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy-thanos" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545318 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy-thanos" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545325 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="config-reloader" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545329 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="config-reloader" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545335 6208 cpu_manager.go:414] "containerMap: removing container" podUID="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" containerName="console" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545339 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" containerName="console" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545382 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy-thanos" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545389 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="config-reloader" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545394 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38365ee-cbdb-4999-a6c9-e78aa4552bb9" containerName="console" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545400 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="prometheus" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545405 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8bfaf2a-e644-40b1-84cd-f2d7d0417207" containerName="console" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545411 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545417 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="thanos-sidecar" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545422 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="kube-rbac-proxy-web" Nov 03 23:30:12.545487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.545432 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" containerName="init-config-reloader" Nov 03 23:30:12.570521 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.570461 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.571958 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.571931 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 23:30:12.575867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.575842 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 23:30:12.575990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.575873 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 23:30:12.575990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.575947 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 23:30:12.576122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576009 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 23:30:12.576122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576090 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 23:30:12.576219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576152 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cfq6h\"" Nov 03 23:30:12.576219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576154 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 23:30:12.576219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576179 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 23:30:12.576407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576220 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 23:30:12.576407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576239 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 23:30:12.576407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576255 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 23:30:12.576407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576356 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 23:30:12.576407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.576406 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 23:30:12.577861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.577843 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 23:30:12.577960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.577869 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-482vvasamrjf7\"" Nov 03 23:30:12.583879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.583859 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 23:30:12.584026 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.584011 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 23:30:12.652088 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652055 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652102 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-web-config\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652178 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652211 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652228 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e831b41a-40d4-4abe-a71c-f30d340c169d-config-out\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652243 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652429 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652326 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e831b41a-40d4-4abe-a71c-f30d340c169d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652429 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652344 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652429 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652413 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652526 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652433 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652526 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652456 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652526 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652473 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652536 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-config\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652561 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652578 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652598 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652640 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dl4f\" (UniqueName: \"kubernetes.io/projected/e831b41a-40d4-4abe-a71c-f30d340c169d-kube-api-access-4dl4f\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652694 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652713 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.652770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.652738 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.753642 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753599 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dl4f\" (UniqueName: \"kubernetes.io/projected/e831b41a-40d4-4abe-a71c-f30d340c169d-kube-api-access-4dl4f\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.753775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753650 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.753775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753674 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.753775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753700 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.753775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753726 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.753775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753749 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-web-config\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753778 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753850 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753892 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e831b41a-40d4-4abe-a71c-f30d340c169d-config-out\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753923 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753958 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e831b41a-40d4-4abe-a71c-f30d340c169d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.753995 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.754058 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.754087 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.754122 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.754165 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.754200 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-config\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.754224 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.754248 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.754283 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.754770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.754583 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.756049 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.755226 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.756049 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.755729 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.756789 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.756704 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e831b41a-40d4-4abe-a71c-f30d340c169d-config-out\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.756894 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.756838 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.757161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.757015 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.757259 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.757239 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.757374 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.757345 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.758102 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.757783 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-web-config\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.758102 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.757900 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.758102 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.757993 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.758560 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.758535 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.758729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.758632 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.758855 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.758746 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.758855 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.758769 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e831b41a-40d4-4abe-a71c-f30d340c169d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.759343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.759324 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.759461 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.759440 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-config\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.760458 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.760437 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e831b41a-40d4-4abe-a71c-f30d340c169d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.760765 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.760747 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e831b41a-40d4-4abe-a71c-f30d340c169d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.766239 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.766218 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dl4f\" (UniqueName: \"kubernetes.io/projected/e831b41a-40d4-4abe-a71c-f30d340c169d-kube-api-access-4dl4f\") pod \"prometheus-k8s-0\" (UID: \"e831b41a-40d4-4abe-a71c-f30d340c169d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:12.945352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:12.945313 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:13.254772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:13.254746 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 23:30:13.254958 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:30:13.254937 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode831b41a_40d4_4abe_a71c_f30d340c169d.slice/crio-92c4fe2550810acfea2b44016dd13b178034d329aca8ea22c6385719b7d680c7 WatchSource:0}: Error finding container 92c4fe2550810acfea2b44016dd13b178034d329aca8ea22c6385719b7d680c7: Status 404 returned error can't find the container with id 92c4fe2550810acfea2b44016dd13b178034d329aca8ea22c6385719b7d680c7 Nov 03 23:30:13.292583 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:13.292551 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f866b162-3fe8-4a69-9ae0-387e0c18d16d" path="/var/lib/kubelet/pods/f866b162-3fe8-4a69-9ae0-387e0c18d16d/volumes" Nov 03 23:30:13.482805 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:13.482771 6208 generic.go:356] "Generic (PLEG): container finished" podID="e831b41a-40d4-4abe-a71c-f30d340c169d" containerID="ca40e0e43af6ee63eba305e24a508aa61d2d9a2bf06eefec094880ca4ebc81ed" exitCode=0 Nov 03 23:30:13.483285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:13.482856 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e831b41a-40d4-4abe-a71c-f30d340c169d","Type":"ContainerDied","Data":"ca40e0e43af6ee63eba305e24a508aa61d2d9a2bf06eefec094880ca4ebc81ed"} Nov 03 23:30:13.483285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:13.482942 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e831b41a-40d4-4abe-a71c-f30d340c169d","Type":"ContainerStarted","Data":"92c4fe2550810acfea2b44016dd13b178034d329aca8ea22c6385719b7d680c7"} Nov 03 23:30:14.490432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:14.490399 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e831b41a-40d4-4abe-a71c-f30d340c169d","Type":"ContainerStarted","Data":"1018237737a5e97494bf0992de6d97173c1264e1250d77ac19a2c970e9a7a860"} Nov 03 23:30:14.490826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:14.490461 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e831b41a-40d4-4abe-a71c-f30d340c169d","Type":"ContainerStarted","Data":"63c9349f07520baf7dbe33f9470998c8ce44cb5c983186788349af735386cf21"} Nov 03 23:30:14.490826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:14.490487 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e831b41a-40d4-4abe-a71c-f30d340c169d","Type":"ContainerStarted","Data":"ec093b5fe4c8994b4de741716b0b9f842fc4b8c4eca0ee36c7bbe1d2a7d8f7d2"} Nov 03 23:30:14.490826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:14.490514 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e831b41a-40d4-4abe-a71c-f30d340c169d","Type":"ContainerStarted","Data":"ecd560aeee143f2581dfa14ad92f2263fa5a606aa4594deb62707e98c5e48809"} Nov 03 23:30:14.490826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:14.490538 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e831b41a-40d4-4abe-a71c-f30d340c169d","Type":"ContainerStarted","Data":"c79f4bd2c496ca84d2709105d0c00838bd1bb38369911d5e3e6f903cd5374a44"} Nov 03 23:30:14.490826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:14.490559 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e831b41a-40d4-4abe-a71c-f30d340c169d","Type":"ContainerStarted","Data":"540cc00630051795d9589d69902570a0ffe9795204fe6876c63b2f30d51562db"} Nov 03 23:30:14.529334 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:14.529287 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.513857644 podStartE2EDuration="2.529273131s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.415487ms" isStatefulPod=false podCreationTimestamp="2026-11-03 23:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:30:14.527670298 +0000 UTC m=+33.350266461" watchObservedRunningTime="2026-11-03 23:30:14.529273131 +0000 UTC m=+33.351869293" Nov 03 23:30:14.874626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:14.874532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:14.975657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:14.975622 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:15.903797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:15.903757 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:17.946045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:17.946013 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:30:32.548235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.548185 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-854967fbd5-4lmhp" podUID="f53faae9-d5e5-4c53-8f4c-1f421e40f203" containerName="console" containerID="cri-o://ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439" gracePeriod=15 Nov 03 23:30:32.788500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.788475 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:30:32.932061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.931965 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca\") pod \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " Nov 03 23:30:32.932061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932018 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle\") pod \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " Nov 03 23:30:32.932061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932053 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj\" (UniqueName: \"kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj\") pod \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " Nov 03 23:30:32.932344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932082 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert\") pod \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " Nov 03 23:30:32.932344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932118 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config\") pod \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " Nov 03 23:30:32.932344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932140 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert\") pod \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " Nov 03 23:30:32.932344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932199 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp\") pod \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " Nov 03 23:30:32.932344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932227 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config\") pod \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\" (UID: \"f53faae9-d5e5-4c53-8f4c-1f421e40f203\") " Nov 03 23:30:32.932616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932492 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle" pod "f53faae9-d5e5-4c53-8f4c-1f421e40f203" (UID: "f53faae9-d5e5-4c53-8f4c-1f421e40f203"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:32.932616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932499 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca" pod "f53faae9-d5e5-4c53-8f4c-1f421e40f203" (UID: "f53faae9-d5e5-4c53-8f4c-1f421e40f203"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:32.932616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932542 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert" pod "f53faae9-d5e5-4c53-8f4c-1f421e40f203" (UID: "f53faae9-d5e5-4c53-8f4c-1f421e40f203"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:32.932751 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932671 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp" pod "f53faae9-d5e5-4c53-8f4c-1f421e40f203" (UID: "f53faae9-d5e5-4c53-8f4c-1f421e40f203"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:30:32.932820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.932796 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config" pod "f53faae9-d5e5-4c53-8f4c-1f421e40f203" (UID: "f53faae9-d5e5-4c53-8f4c-1f421e40f203"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:30:32.934278 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.934257 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj" pod "f53faae9-d5e5-4c53-8f4c-1f421e40f203" (UID: "f53faae9-d5e5-4c53-8f4c-1f421e40f203"). InnerVolumeSpecName "kube-api-access-z44cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:30:32.934743 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.934723 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config" pod "f53faae9-d5e5-4c53-8f4c-1f421e40f203" (UID: "f53faae9-d5e5-4c53-8f4c-1f421e40f203"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:32.934838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:32.934810 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert" pod "f53faae9-d5e5-4c53-8f4c-1f421e40f203" (UID: "f53faae9-d5e5-4c53-8f4c-1f421e40f203"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:30:33.032848 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.032810 6208 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-oauth-serving-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:33.032848 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.032845 6208 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-config\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:33.032848 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.032856 6208 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-serving-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:33.033065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.032865 6208 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53faae9-d5e5-4c53-8f4c-1f421e40f203-tmp\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:33.033065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.032874 6208 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f53faae9-d5e5-4c53-8f4c-1f421e40f203-console-oauth-config\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:33.033065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.032883 6208 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:33.033065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.032893 6208 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53faae9-d5e5-4c53-8f4c-1f421e40f203-trusted-ca-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:33.033065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.032902 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z44cj\" (UniqueName: \"kubernetes.io/projected/f53faae9-d5e5-4c53-8f4c-1f421e40f203-kube-api-access-z44cj\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:30:33.555428 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.555338 6208 generic.go:356] "Generic (PLEG): container finished" podID="f53faae9-d5e5-4c53-8f4c-1f421e40f203" containerID="ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439" exitCode=0 Nov 03 23:30:33.555872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.555426 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854967fbd5-4lmhp" Nov 03 23:30:33.555872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.555423 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854967fbd5-4lmhp" event={"ID":"f53faae9-d5e5-4c53-8f4c-1f421e40f203","Type":"ContainerDied","Data":"ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439"} Nov 03 23:30:33.555872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.555584 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854967fbd5-4lmhp" event={"ID":"f53faae9-d5e5-4c53-8f4c-1f421e40f203","Type":"ContainerDied","Data":"38ce56ebdf0ca36e6f651f549321a729bcd596b712616053e74b8ca532d613c3"} Nov 03 23:30:33.555872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.555615 6208 scope.go:122] "RemoveContainer" containerID="ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439" Nov 03 23:30:33.564078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.564048 6208 scope.go:122] "RemoveContainer" containerID="ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439" Nov 03 23:30:33.564362 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:33.564341 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439\": container with ID starting with ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439 not found: ID does not exist" containerID="ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439" Nov 03 23:30:33.564444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.564374 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439"} err="failed to get container status \"ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439\": rpc error: code = NotFound desc = could not find container \"ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439\": container with ID starting with ff0a95c5b431185a2a1f89f6eb9e6f7b9e477d404ff9b8d5d9297aa6ac0bf439 not found: ID does not exist" Nov 03 23:30:33.576130 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.576107 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854967fbd5-4lmhp"] Nov 03 23:30:33.580517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:33.580496 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-854967fbd5-4lmhp"] Nov 03 23:30:35.291068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:35.291016 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53faae9-d5e5-4c53-8f4c-1f421e40f203" path="/var/lib/kubelet/pods/f53faae9-d5e5-4c53-8f4c-1f421e40f203/volumes" Nov 03 23:30:35.702102 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:30:35.702005 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:30:40.972878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:40.972846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:41.066214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:41.066179 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:44.667359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:44.667325 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:47.506189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:30:47.506143 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:31:00.735933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:31:00.735894 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:31:00.831221 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:31:00.831190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:31:05.701968 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:31:05.701921 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:31:07.264125 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:31:07.264074 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:31:07.280162 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:31:07.280131 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:31:07.678694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:31:07.678618 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 23:31:35.702425 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:31:35.702344 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:31:44.729929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:31:44.729866 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:31:45.251552 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:31:45.251488 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:31:45.351340 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:31:45.351300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:31:45.445206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:31:45.445172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:05.701679 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:32:05.701644 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:32:24.311655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:24.311615 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:24.519550 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:24.519515 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:24.662041 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:24.661945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:26.227707 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.227677 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:26.318144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.318112 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4"] Nov 03 23:32:26.318457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.318433 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f53faae9-d5e5-4c53-8f4c-1f421e40f203" containerName="console" Nov 03 23:32:26.318457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.318452 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53faae9-d5e5-4c53-8f4c-1f421e40f203" containerName="console" Nov 03 23:32:26.318605 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.318501 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53faae9-d5e5-4c53-8f4c-1f421e40f203" containerName="console" Nov 03 23:32:26.321465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.321447 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.332495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.332471 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4"] Nov 03 23:32:26.341075 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.341058 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 23:32:26.341075 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.341066 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 23:32:26.342346 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.342334 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-6j4q7\"" Nov 03 23:32:26.372286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.372266 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.372374 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.372312 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.372421 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.372378 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc9v4\" (UniqueName: \"kubernetes.io/projected/90904130-60ec-4811-b110-4ed57f4ad6a7-kube-api-access-fc9v4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.388960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.388935 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:26.473568 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.473549 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.473656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.473579 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9v4\" (UniqueName: \"kubernetes.io/projected/90904130-60ec-4811-b110-4ed57f4ad6a7-kube-api-access-fc9v4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.473656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.473614 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.473922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.473904 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.473956 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.473932 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.486316 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.486264 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9v4\" (UniqueName: \"kubernetes.io/projected/90904130-60ec-4811-b110-4ed57f4ad6a7-kube-api-access-fc9v4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.593291 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.593261 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86"] Nov 03 23:32:26.596594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.596579 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.601515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.601501 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-m64nv\"" Nov 03 23:32:26.610141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.610116 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86"] Nov 03 23:32:26.637153 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.637126 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:32:26.674820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.674793 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.674945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.674827 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.674945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.674862 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kp7\" (UniqueName: \"kubernetes.io/projected/ba31bff8-3bbf-48a2-b004-83a041cbc695-kube-api-access-k5kp7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.775929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.775855 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.775929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.775891 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.776143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.775942 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kp7\" (UniqueName: \"kubernetes.io/projected/ba31bff8-3bbf-48a2-b004-83a041cbc695-kube-api-access-k5kp7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.776330 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.776309 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.776371 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.776319 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.788332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.788308 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kp7\" (UniqueName: \"kubernetes.io/projected/ba31bff8-3bbf-48a2-b004-83a041cbc695-kube-api-access-k5kp7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.911596 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.911566 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:32:26.911832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.911810 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4"] Nov 03 23:32:26.915120 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:32:26.915090 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90904130_60ec_4811_b110_4ed57f4ad6a7.slice/crio-e39cbacec01d3edd31a0edc02a0a8dcd7403b91b319921c364fb51ea8195316d WatchSource:0}: Error finding container e39cbacec01d3edd31a0edc02a0a8dcd7403b91b319921c364fb51ea8195316d: Status 404 returned error can't find the container with id e39cbacec01d3edd31a0edc02a0a8dcd7403b91b319921c364fb51ea8195316d Nov 03 23:32:26.919298 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:26.919278 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:32:27.185587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:27.185519 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86"] Nov 03 23:32:27.188943 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:32:27.188918 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba31bff8_3bbf_48a2_b004_83a041cbc695.slice/crio-3a76d428ed59d17652d5f90c23c010630698ad2c60852d97586693cd5bfc53d6 WatchSource:0}: Error finding container 3a76d428ed59d17652d5f90c23c010630698ad2c60852d97586693cd5bfc53d6: Status 404 returned error can't find the container with id 3a76d428ed59d17652d5f90c23c010630698ad2c60852d97586693cd5bfc53d6 Nov 03 23:32:27.652578 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:27.652540 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:27.917352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:27.917274 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" event={"ID":"ba31bff8-3bbf-48a2-b004-83a041cbc695","Type":"ContainerStarted","Data":"3a76d428ed59d17652d5f90c23c010630698ad2c60852d97586693cd5bfc53d6"} Nov 03 23:32:27.918684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:27.918656 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" event={"ID":"90904130-60ec-4811-b110-4ed57f4ad6a7","Type":"ContainerStarted","Data":"e39cbacec01d3edd31a0edc02a0a8dcd7403b91b319921c364fb51ea8195316d"} Nov 03 23:32:28.092912 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.092868 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz"] Nov 03 23:32:28.096336 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.096311 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.102067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.102011 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-vmdgx\"" Nov 03 23:32:28.107679 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.107357 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz"] Nov 03 23:32:28.188932 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.188857 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49k5\" (UniqueName: \"kubernetes.io/projected/11c3134b-be4d-4694-afc5-04575ec660fe-kube-api-access-p49k5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.188932 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.188897 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.189186 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.189005 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.289547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.289507 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.289737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.289597 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49k5\" (UniqueName: \"kubernetes.io/projected/11c3134b-be4d-4694-afc5-04575ec660fe-kube-api-access-p49k5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.289737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.289626 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.289972 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.289947 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.290019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.289963 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.303299 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.303272 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49k5\" (UniqueName: \"kubernetes.io/projected/11c3134b-be4d-4694-afc5-04575ec660fe-kube-api-access-p49k5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.416614 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.416579 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:32:28.635180 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.635148 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s"] Nov 03 23:32:28.638482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.638459 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.643215 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.643188 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-mcr2p\"" Nov 03 23:32:28.648824 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.648799 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s"] Nov 03 23:32:28.693423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.693387 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkth8\" (UniqueName: \"kubernetes.io/projected/8fe95351-fb90-49ed-8347-ac51bf4c03c5-kube-api-access-bkth8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.693880 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.693442 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.693880 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.693488 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.702695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.702660 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz"] Nov 03 23:32:28.706077 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:32:28.706043 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c3134b_be4d_4694_afc5_04575ec660fe.slice/crio-ff7bda452d827b986e75e7f31ad0412da0d5ab952e96417dfd2f7bd83c8e5f67 WatchSource:0}: Error finding container ff7bda452d827b986e75e7f31ad0412da0d5ab952e96417dfd2f7bd83c8e5f67: Status 404 returned error can't find the container with id ff7bda452d827b986e75e7f31ad0412da0d5ab952e96417dfd2f7bd83c8e5f67 Nov 03 23:32:28.733421 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.733365 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb"] Nov 03 23:32:28.737478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.737109 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.742346 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.742322 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-zdzks\"" Nov 03 23:32:28.747776 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.747751 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb"] Nov 03 23:32:28.795362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.794347 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkth8\" (UniqueName: \"kubernetes.io/projected/8fe95351-fb90-49ed-8347-ac51bf4c03c5-kube-api-access-bkth8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.795362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.794401 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhb4\" (UniqueName: \"kubernetes.io/projected/92e5c088-8670-4d9c-ad61-7e5ce4e55649-kube-api-access-zvhb4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.795362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.794433 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.795362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.794478 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.795362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.794512 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.795362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.794581 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.795362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.795029 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.795362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.795284 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.807706 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.807681 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkth8\" (UniqueName: \"kubernetes.io/projected/8fe95351-fb90-49ed-8347-ac51bf4c03c5-kube-api-access-bkth8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.836132 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.836102 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n"] Nov 03 23:32:28.839630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.839608 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:28.843766 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.843746 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-flfwh\"" Nov 03 23:32:28.849487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.849465 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n"] Nov 03 23:32:28.895591 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.895551 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.895729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.895616 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:28.895729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.895654 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qm4\" (UniqueName: \"kubernetes.io/projected/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-kube-api-access-m5qm4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:28.895729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.895698 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.895864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.895831 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:28.895915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.895905 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhb4\" (UniqueName: \"kubernetes.io/projected/92e5c088-8670-4d9c-ad61-7e5ce4e55649-kube-api-access-zvhb4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.896050 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.896005 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.896163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.896014 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.909096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.909067 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhb4\" (UniqueName: \"kubernetes.io/projected/92e5c088-8670-4d9c-ad61-7e5ce4e55649-kube-api-access-zvhb4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:28.923789 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.923759 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" event={"ID":"11c3134b-be4d-4694-afc5-04575ec660fe","Type":"ContainerStarted","Data":"ff7bda452d827b986e75e7f31ad0412da0d5ab952e96417dfd2f7bd83c8e5f67"} Nov 03 23:32:28.930011 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.929978 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np"] Nov 03 23:32:28.933863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.933496 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:28.938120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.938097 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-tbj5g\"" Nov 03 23:32:28.944619 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.944593 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np"] Nov 03 23:32:28.961368 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.961349 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:32:28.997136 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.997057 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:28.997136 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.997107 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:28.997327 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.997206 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qm4\" (UniqueName: \"kubernetes.io/projected/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-kube-api-access-m5qm4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:28.997327 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.997275 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:28.997327 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.997314 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55r9g\" (UniqueName: \"kubernetes.io/projected/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-kube-api-access-55r9g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:28.997472 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.997344 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:28.997953 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.997932 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:28.998224 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:28.998191 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:29.013897 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.013837 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qm4\" (UniqueName: \"kubernetes.io/projected/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-kube-api-access-m5qm4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:29.066650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.066382 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:32:29.098221 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.098191 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:29.098365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.098238 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55r9g\" (UniqueName: \"kubernetes.io/projected/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-kube-api-access-55r9g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:29.098365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.098300 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:29.098680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.098656 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:29.098855 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.098839 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:29.112257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.111843 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55r9g\" (UniqueName: \"kubernetes.io/projected/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-kube-api-access-55r9g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:29.164186 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.164155 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:32:29.250519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.250416 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s"] Nov 03 23:32:29.264908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.264888 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:32:29.360770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.360739 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb"] Nov 03 23:32:29.456553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.456459 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n"] Nov 03 23:32:29.459789 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:32:29.459759 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e98d1d6_dc5d_4571_954e_5700f7cf2f7b.slice/crio-695b4c134ef5b3d2bd4d3b96edad5018eb38896ca63332e63e47f68c39238933 WatchSource:0}: Error finding container 695b4c134ef5b3d2bd4d3b96edad5018eb38896ca63332e63e47f68c39238933: Status 404 returned error can't find the container with id 695b4c134ef5b3d2bd4d3b96edad5018eb38896ca63332e63e47f68c39238933 Nov 03 23:32:29.559665 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.559591 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np"] Nov 03 23:32:29.565227 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:32:29.565186 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6e8876_231d_4e24_a4cc_f60fb0db11d5.slice/crio-9b7fcb7b1f749c10c216f00f50deda2e747fc6a4ff267f43bb82fb5baaa66989 WatchSource:0}: Error finding container 9b7fcb7b1f749c10c216f00f50deda2e747fc6a4ff267f43bb82fb5baaa66989: Status 404 returned error can't find the container with id 9b7fcb7b1f749c10c216f00f50deda2e747fc6a4ff267f43bb82fb5baaa66989 Nov 03 23:32:29.931833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.931744 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" event={"ID":"8fe95351-fb90-49ed-8347-ac51bf4c03c5","Type":"ContainerStarted","Data":"e16d3a80469af59c4664c9d8fe8db3362fb2ad7a1d48b0f5c219931b18d97424"} Nov 03 23:32:29.937616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.933737 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" event={"ID":"1b6e8876-231d-4e24-a4cc-f60fb0db11d5","Type":"ContainerStarted","Data":"9b7fcb7b1f749c10c216f00f50deda2e747fc6a4ff267f43bb82fb5baaa66989"} Nov 03 23:32:29.937616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.935136 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" event={"ID":"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b","Type":"ContainerStarted","Data":"695b4c134ef5b3d2bd4d3b96edad5018eb38896ca63332e63e47f68c39238933"} Nov 03 23:32:29.937616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:29.937024 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" event={"ID":"92e5c088-8670-4d9c-ad61-7e5ce4e55649","Type":"ContainerStarted","Data":"218b6b3c1693f9c4fba20c52ed3fc666eb05743e08df73bd9c74c8b699c9e5f3"} Nov 03 23:32:30.551540 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:30.551487 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:30.658462 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:30.658398 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:31.757229 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:31.757110 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:31.869599 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:31.869568 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:31.980794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:31.980739 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:32.654018 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.653976 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6"] Nov 03 23:32:32.659104 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.658887 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.669166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.668912 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6"] Nov 03 23:32:32.735711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.735665 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.735973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.735730 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.735973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.735812 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mklmt\" (UniqueName: \"kubernetes.io/projected/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-kube-api-access-mklmt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.836774 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.836730 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.837394 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.836780 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.837394 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.836855 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mklmt\" (UniqueName: \"kubernetes.io/projected/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-kube-api-access-mklmt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.837542 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.837438 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.837724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.837701 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.852022 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.851996 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mklmt\" (UniqueName: \"kubernetes.io/projected/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-kube-api-access-mklmt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:32.987209 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:32.986914 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:32:33.356735 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:33.356614 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6"] Nov 03 23:32:33.364389 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:32:33.364156 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode063a0d5_49bc_49f6_9f30_25ecd024ea4d.slice/crio-cf1552906b44095bcc3b4337d44ca1c3a430a29f59dd038cea7049fa1cb9e7ff WatchSource:0}: Error finding container cf1552906b44095bcc3b4337d44ca1c3a430a29f59dd038cea7049fa1cb9e7ff: Status 404 returned error can't find the container with id cf1552906b44095bcc3b4337d44ca1c3a430a29f59dd038cea7049fa1cb9e7ff Nov 03 23:32:33.974801 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:33.974755 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" event={"ID":"e063a0d5-49bc-49f6-9f30-25ecd024ea4d","Type":"ContainerStarted","Data":"cf1552906b44095bcc3b4337d44ca1c3a430a29f59dd038cea7049fa1cb9e7ff"} Nov 03 23:32:34.842274 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:34.842236 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:35.769351 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:32:35.769306 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:32:41.039012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:41.037832 6208 generic.go:356] "Generic (PLEG): container finished" podID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerID="c5e04f3957f3709a4ec163cca44d0e0c8aa28bfd927c2875694c9c6c55ea01f8" exitCode=0 Nov 03 23:32:41.039012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:41.038005 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" event={"ID":"90904130-60ec-4811-b110-4ed57f4ad6a7","Type":"ContainerDied","Data":"c5e04f3957f3709a4ec163cca44d0e0c8aa28bfd927c2875694c9c6c55ea01f8"} Nov 03 23:32:41.047017 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:41.046985 6208 generic.go:356] "Generic (PLEG): container finished" podID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerID="1dfd6956aa0d9b98696b0435440367d192f57f84cde91ef90593424ef1cba490" exitCode=0 Nov 03 23:32:41.047017 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:41.047076 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" event={"ID":"ba31bff8-3bbf-48a2-b004-83a041cbc695","Type":"ContainerDied","Data":"1dfd6956aa0d9b98696b0435440367d192f57f84cde91ef90593424ef1cba490"} Nov 03 23:32:43.060256 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:43.060216 6208 generic.go:356] "Generic (PLEG): container finished" podID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerID="9dbeec1b93cc2e44a5e3163ee24f40fbf0885d59d3701937a18efa67d0f77542" exitCode=0 Nov 03 23:32:43.060729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:43.060313 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" event={"ID":"ba31bff8-3bbf-48a2-b004-83a041cbc695","Type":"ContainerDied","Data":"9dbeec1b93cc2e44a5e3163ee24f40fbf0885d59d3701937a18efa67d0f77542"} Nov 03 23:32:43.064307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:43.064278 6208 generic.go:356] "Generic (PLEG): container finished" podID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerID="c40c81bb5558eef40ac0a96db404f7e8d27f249f28a67033c226e5680ed0f7a3" exitCode=0 Nov 03 23:32:43.064412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:43.064349 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" event={"ID":"e063a0d5-49bc-49f6-9f30-25ecd024ea4d","Type":"ContainerDied","Data":"c40c81bb5558eef40ac0a96db404f7e8d27f249f28a67033c226e5680ed0f7a3"} Nov 03 23:32:44.069276 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:44.069241 6208 generic.go:356] "Generic (PLEG): container finished" podID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerID="abbcb3e2efc52d55edbda50f4868d858c1abbb24c3ae66c60a556e73ccff2287" exitCode=0 Nov 03 23:32:44.069694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:44.069325 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" event={"ID":"e063a0d5-49bc-49f6-9f30-25ecd024ea4d","Type":"ContainerDied","Data":"abbcb3e2efc52d55edbda50f4868d858c1abbb24c3ae66c60a556e73ccff2287"} Nov 03 23:32:44.077854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:44.077821 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" event={"ID":"92e5c088-8670-4d9c-ad61-7e5ce4e55649","Type":"ContainerStarted","Data":"74771c166df33006e05e5b4772b06ba97ffeace0ece0e299d2e3b882a705202a"} Nov 03 23:32:44.080132 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:44.080103 6208 generic.go:356] "Generic (PLEG): container finished" podID="11c3134b-be4d-4694-afc5-04575ec660fe" containerID="f886d39ce757240696cb05341730ad4198d745371f253f0fb516176d4595e9fe" exitCode=0 Nov 03 23:32:44.080242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:44.080146 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" event={"ID":"11c3134b-be4d-4694-afc5-04575ec660fe","Type":"ContainerDied","Data":"f886d39ce757240696cb05341730ad4198d745371f253f0fb516176d4595e9fe"} Nov 03 23:32:44.788773 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:44.788731 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:45.089452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:45.089418 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" event={"ID":"90904130-60ec-4811-b110-4ed57f4ad6a7","Type":"ContainerStarted","Data":"6664b3dc8defcd480ab9ac92aa2a1ef1c952c4dacbdc032219efdab39c156d11"} Nov 03 23:32:45.092344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:45.092319 6208 generic.go:356] "Generic (PLEG): container finished" podID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerID="977a6af92e0ecd4e1c85e98082ab3c0ea4d00b0c37a30d682162d108ae3bd278" exitCode=0 Nov 03 23:32:45.092583 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:45.092526 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" event={"ID":"8fe95351-fb90-49ed-8347-ac51bf4c03c5","Type":"ContainerDied","Data":"977a6af92e0ecd4e1c85e98082ab3c0ea4d00b0c37a30d682162d108ae3bd278"} Nov 03 23:32:45.096058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:45.095480 6208 generic.go:356] "Generic (PLEG): container finished" podID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerID="e21fb460d6ae8c635ce76a8760806a3fb850c67a467b92dca48eba0785d04d35" exitCode=0 Nov 03 23:32:45.096058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:45.095562 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" event={"ID":"1b6e8876-231d-4e24-a4cc-f60fb0db11d5","Type":"ContainerDied","Data":"e21fb460d6ae8c635ce76a8760806a3fb850c67a467b92dca48eba0785d04d35"} Nov 03 23:32:45.099989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:45.099301 6208 generic.go:356] "Generic (PLEG): container finished" podID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerID="b1ebd1d8fdc9066befafd96971089164d088e7982a13353e027f573b31d09712" exitCode=0 Nov 03 23:32:45.099989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:45.099367 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" event={"ID":"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b","Type":"ContainerDied","Data":"b1ebd1d8fdc9066befafd96971089164d088e7982a13353e027f573b31d09712"} Nov 03 23:32:45.101822 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:45.101799 6208 generic.go:356] "Generic (PLEG): container finished" podID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerID="74771c166df33006e05e5b4772b06ba97ffeace0ece0e299d2e3b882a705202a" exitCode=0 Nov 03 23:32:45.101906 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:45.101873 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" event={"ID":"92e5c088-8670-4d9c-ad61-7e5ce4e55649","Type":"ContainerDied","Data":"74771c166df33006e05e5b4772b06ba97ffeace0ece0e299d2e3b882a705202a"} Nov 03 23:32:46.113068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:46.112684 6208 generic.go:356] "Generic (PLEG): container finished" podID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerID="6664b3dc8defcd480ab9ac92aa2a1ef1c952c4dacbdc032219efdab39c156d11" exitCode=0 Nov 03 23:32:46.113068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:46.112799 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" event={"ID":"90904130-60ec-4811-b110-4ed57f4ad6a7","Type":"ContainerDied","Data":"6664b3dc8defcd480ab9ac92aa2a1ef1c952c4dacbdc032219efdab39c156d11"} Nov 03 23:32:47.123893 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:47.123854 6208 generic.go:356] "Generic (PLEG): container finished" podID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerID="0e901cb037e3c3717958425b25a7234003709a14a1a819e74213fb730464d363" exitCode=0 Nov 03 23:32:47.124361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:47.124107 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" event={"ID":"8fe95351-fb90-49ed-8347-ac51bf4c03c5","Type":"ContainerDied","Data":"0e901cb037e3c3717958425b25a7234003709a14a1a819e74213fb730464d363"} Nov 03 23:32:47.127660 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:47.127636 6208 generic.go:356] "Generic (PLEG): container finished" podID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerID="bebc142ddfbc80d188837bc36ce5c9bdea96c05bb0f8b1b97e63bcdcdd10286e" exitCode=0 Nov 03 23:32:47.127759 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:47.127740 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" event={"ID":"1b6e8876-231d-4e24-a4cc-f60fb0db11d5","Type":"ContainerDied","Data":"bebc142ddfbc80d188837bc36ce5c9bdea96c05bb0f8b1b97e63bcdcdd10286e"} Nov 03 23:32:47.133737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:47.133712 6208 generic.go:356] "Generic (PLEG): container finished" podID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerID="aab294efbcb37a8d26729afe48f46cf6276e43018ccf2afbcbb9ceee7e158da5" exitCode=0 Nov 03 23:32:47.133835 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:47.133787 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" event={"ID":"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b","Type":"ContainerDied","Data":"aab294efbcb37a8d26729afe48f46cf6276e43018ccf2afbcbb9ceee7e158da5"} Nov 03 23:32:47.136558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:47.136535 6208 generic.go:356] "Generic (PLEG): container finished" podID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerID="b0d3ef274cc1cb91e1eddbb7ee6e463c8d895139805143aa537bc52212530b1a" exitCode=0 Nov 03 23:32:47.136631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:47.136609 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" event={"ID":"92e5c088-8670-4d9c-ad61-7e5ce4e55649","Type":"ContainerDied","Data":"b0d3ef274cc1cb91e1eddbb7ee6e463c8d895139805143aa537bc52212530b1a"} Nov 03 23:32:48.148542 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:48.148502 6208 generic.go:356] "Generic (PLEG): container finished" podID="11c3134b-be4d-4694-afc5-04575ec660fe" containerID="4aaf18aaba5e13935bbd215b6f5b028bcd4c5f68c3457c10b4eb6297c657b0c9" exitCode=0 Nov 03 23:32:48.148931 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:48.148585 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" event={"ID":"11c3134b-be4d-4694-afc5-04575ec660fe","Type":"ContainerDied","Data":"4aaf18aaba5e13935bbd215b6f5b028bcd4c5f68c3457c10b4eb6297c657b0c9"} Nov 03 23:32:52.809002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:52.808892 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:54.619516 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:54.619476 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:57.279347 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:57.279209 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:57.406360 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:57.406205 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:59.244992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:59.244942 6208 generic.go:356] "Generic (PLEG): container finished" podID="11c3134b-be4d-4694-afc5-04575ec660fe" containerID="022ff4c1d31b692ab9dfb349ae00273c2546ef439815728a81d9a2d1b0af95d4" exitCode=0 Nov 03 23:32:59.245410 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:59.245149 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" event={"ID":"11c3134b-be4d-4694-afc5-04575ec660fe","Type":"ContainerDied","Data":"022ff4c1d31b692ab9dfb349ae00273c2546ef439815728a81d9a2d1b0af95d4"} Nov 03 23:32:59.249358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:59.248548 6208 generic.go:356] "Generic (PLEG): container finished" podID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerID="70670afc74985acbff239197b14841b903afb2a45072f2cc46eed7f6ea48f75e" exitCode=0 Nov 03 23:32:59.249358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:59.248691 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" event={"ID":"ba31bff8-3bbf-48a2-b004-83a041cbc695","Type":"ContainerDied","Data":"70670afc74985acbff239197b14841b903afb2a45072f2cc46eed7f6ea48f75e"} Nov 03 23:32:59.252001 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:59.251983 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:32:59.252989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:59.252860 6208 generic.go:356] "Generic (PLEG): container finished" podID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerID="9a8b962e9912e829a7a2c2676f4e169d1203445f8ab0200332b2451120abc96b" exitCode=1 Nov 03 23:32:59.252989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:32:59.252933 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" event={"ID":"e063a0d5-49bc-49f6-9f30-25ecd024ea4d","Type":"ContainerDied","Data":"9a8b962e9912e829a7a2c2676f4e169d1203445f8ab0200332b2451120abc96b"} Nov 03 23:33:00.449682 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.449327 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:33:00.501707 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.501000 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:33:00.513578 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.513509 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:33:00.530027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.529925 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:33:00.535192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.534536 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c3134b-be4d-4694-afc5-04575ec660fe-kube-api-access-p49k5\" (UniqueName: \"kubernetes.io/projected/11c3134b-be4d-4694-afc5-04575ec660fe-kube-api-access-p49k5\") pod \"11c3134b-be4d-4694-afc5-04575ec660fe\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " Nov 03 23:33:00.535192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.534587 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-bundle\") pod \"ba31bff8-3bbf-48a2-b004-83a041cbc695\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " Nov 03 23:33:00.535192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.534669 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba31bff8-3bbf-48a2-b004-83a041cbc695-kube-api-access-k5kp7\" (UniqueName: \"kubernetes.io/projected/ba31bff8-3bbf-48a2-b004-83a041cbc695-kube-api-access-k5kp7\") pod \"ba31bff8-3bbf-48a2-b004-83a041cbc695\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " Nov 03 23:33:00.535192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.534730 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-util\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-util\") pod \"ba31bff8-3bbf-48a2-b004-83a041cbc695\" (UID: \"ba31bff8-3bbf-48a2-b004-83a041cbc695\") " Nov 03 23:33:00.535192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.534760 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-util\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-util\") pod \"11c3134b-be4d-4694-afc5-04575ec660fe\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " Nov 03 23:33:00.535192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.534804 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-bundle\") pod \"11c3134b-be4d-4694-afc5-04575ec660fe\" (UID: \"11c3134b-be4d-4694-afc5-04575ec660fe\") " Nov 03 23:33:00.536343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.536286 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-bundle" pod "11c3134b-be4d-4694-afc5-04575ec660fe" (UID: "11c3134b-be4d-4694-afc5-04575ec660fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:00.541618 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.541589 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-util" pod "11c3134b-be4d-4694-afc5-04575ec660fe" (UID: "11c3134b-be4d-4694-afc5-04575ec660fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:00.542232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.542196 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-bundle" pod "ba31bff8-3bbf-48a2-b004-83a041cbc695" (UID: "ba31bff8-3bbf-48a2-b004-83a041cbc695"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:00.542396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.542367 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-util" pod "ba31bff8-3bbf-48a2-b004-83a041cbc695" (UID: "ba31bff8-3bbf-48a2-b004-83a041cbc695"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:00.544509 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.544470 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba31bff8-3bbf-48a2-b004-83a041cbc695-kube-api-access-k5kp7" pod "ba31bff8-3bbf-48a2-b004-83a041cbc695" (UID: "ba31bff8-3bbf-48a2-b004-83a041cbc695"). InnerVolumeSpecName "kube-api-access-k5kp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:00.546514 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.546423 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c3134b-be4d-4694-afc5-04575ec660fe-kube-api-access-p49k5" pod "11c3134b-be4d-4694-afc5-04575ec660fe" (UID: "11c3134b-be4d-4694-afc5-04575ec660fe"). InnerVolumeSpecName "kube-api-access-p49k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:00.635541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.635482 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-kube-api-access-mklmt\" (UniqueName: \"kubernetes.io/projected/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-kube-api-access-mklmt\") pod \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " Nov 03 23:33:00.635541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.635542 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-bundle\") pod \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " Nov 03 23:33:00.635783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.635602 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-util\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-util\") pod \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\" (UID: \"e063a0d5-49bc-49f6-9f30-25ecd024ea4d\") " Nov 03 23:33:00.635836 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.635811 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5kp7\" (UniqueName: \"kubernetes.io/projected/ba31bff8-3bbf-48a2-b004-83a041cbc695-kube-api-access-k5kp7\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:00.635836 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.635829 6208 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-util\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:00.635904 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.635846 6208 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-util\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:00.635904 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.635860 6208 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c3134b-be4d-4694-afc5-04575ec660fe-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:00.635904 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.635874 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p49k5\" (UniqueName: \"kubernetes.io/projected/11c3134b-be4d-4694-afc5-04575ec660fe-kube-api-access-p49k5\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:00.635904 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.635888 6208 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba31bff8-3bbf-48a2-b004-83a041cbc695-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:00.637012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.636985 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-bundle" pod "e063a0d5-49bc-49f6-9f30-25ecd024ea4d" (UID: "e063a0d5-49bc-49f6-9f30-25ecd024ea4d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:00.642013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.641982 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-util" pod "e063a0d5-49bc-49f6-9f30-25ecd024ea4d" (UID: "e063a0d5-49bc-49f6-9f30-25ecd024ea4d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:00.643694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.643658 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-kube-api-access-mklmt" pod "e063a0d5-49bc-49f6-9f30-25ecd024ea4d" (UID: "e063a0d5-49bc-49f6-9f30-25ecd024ea4d"). InnerVolumeSpecName "kube-api-access-mklmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:00.737146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.737068 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mklmt\" (UniqueName: \"kubernetes.io/projected/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-kube-api-access-mklmt\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:00.737146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.737103 6208 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:00.737146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:00.737118 6208 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e063a0d5-49bc-49f6-9f30-25ecd024ea4d-util\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:01.266095 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.266057 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" event={"ID":"11c3134b-be4d-4694-afc5-04575ec660fe","Type":"ContainerDied","Data":"ff7bda452d827b986e75e7f31ad0412da0d5ab952e96417dfd2f7bd83c8e5f67"} Nov 03 23:33:01.266294 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.266152 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff7bda452d827b986e75e7f31ad0412da0d5ab952e96417dfd2f7bd83c8e5f67" Nov 03 23:33:01.266294 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.266158 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qrfmz" Nov 03 23:33:01.268231 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.268208 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" Nov 03 23:33:01.268635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.268611 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7158k86" event={"ID":"ba31bff8-3bbf-48a2-b004-83a041cbc695","Type":"ContainerDied","Data":"3a76d428ed59d17652d5f90c23c010630698ad2c60852d97586693cd5bfc53d6"} Nov 03 23:33:01.268714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.268659 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a76d428ed59d17652d5f90c23c010630698ad2c60852d97586693cd5bfc53d6" Nov 03 23:33:01.270087 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.270065 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:33:01.276683 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.276660 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" event={"ID":"e063a0d5-49bc-49f6-9f30-25ecd024ea4d","Type":"ContainerDied","Data":"cf1552906b44095bcc3b4337d44ca1c3a430a29f59dd038cea7049fa1cb9e7ff"} Nov 03 23:33:01.276811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.276703 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf1552906b44095bcc3b4337d44ca1c3a430a29f59dd038cea7049fa1cb9e7ff" Nov 03 23:33:01.276811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.276706 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6" Nov 03 23:33:01.965286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:01.965253 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:02.155847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.155782 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:02.284742 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.284657 6208 generic.go:356] "Generic (PLEG): container finished" podID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerID="8d0a7c32a07c7eeb36c1bf2bad56ee303481c70f0966116d9ca92909968cd3e2" exitCode=0 Nov 03 23:33:02.284742 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.284711 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" event={"ID":"90904130-60ec-4811-b110-4ed57f4ad6a7","Type":"ContainerDied","Data":"8d0a7c32a07c7eeb36c1bf2bad56ee303481c70f0966116d9ca92909968cd3e2"} Nov 03 23:33:02.319585 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.319552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:02.539397 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.539342 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:02.961582 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961556 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg"] Nov 03 23:33:02.961852 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961839 6208 cpu_manager.go:414] "containerMap: removing container" podUID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerName="extract" Nov 03 23:33:02.961895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961855 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerName="extract" Nov 03 23:33:02.961895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961868 6208 cpu_manager.go:414] "containerMap: removing container" podUID="11c3134b-be4d-4694-afc5-04575ec660fe" containerName="extract" Nov 03 23:33:02.961895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961875 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c3134b-be4d-4694-afc5-04575ec660fe" containerName="extract" Nov 03 23:33:02.961895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961891 6208 cpu_manager.go:414] "containerMap: removing container" podUID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerName="pull" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961899 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerName="pull" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961917 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerName="util" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961923 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerName="util" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961951 6208 cpu_manager.go:414] "containerMap: removing container" podUID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerName="util" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961958 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerName="util" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961971 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerName="pull" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961978 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerName="pull" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.961999 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerName="extract" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.962006 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerName="extract" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.962018 6208 cpu_manager.go:414] "containerMap: removing container" podUID="11c3134b-be4d-4694-afc5-04575ec660fe" containerName="pull" Nov 03 23:33:02.962024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.962025 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c3134b-be4d-4694-afc5-04575ec660fe" containerName="pull" Nov 03 23:33:02.962475 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.962051 6208 cpu_manager.go:414] "containerMap: removing container" podUID="11c3134b-be4d-4694-afc5-04575ec660fe" containerName="util" Nov 03 23:33:02.962475 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.962058 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c3134b-be4d-4694-afc5-04575ec660fe" containerName="util" Nov 03 23:33:02.962475 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.962138 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba31bff8-3bbf-48a2-b004-83a041cbc695" containerName="extract" Nov 03 23:33:02.962475 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.962148 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c3134b-be4d-4694-afc5-04575ec660fe" containerName="extract" Nov 03 23:33:02.962475 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.962164 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e063a0d5-49bc-49f6-9f30-25ecd024ea4d" containerName="extract" Nov 03 23:33:02.975191 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.975165 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg"] Nov 03 23:33:02.975579 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.975275 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:02.984690 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.984669 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 23:33:02.985070 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.985056 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 23:33:02.985154 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:02.985139 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-crjjz\"" Nov 03 23:33:03.054772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.054717 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63ac6df1-998d-43e6-bf49-2aeff97e4a4b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-c5hhg\" (UID: \"63ac6df1-998d-43e6-bf49-2aeff97e4a4b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:03.054772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.054747 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlb5t\" (UniqueName: \"kubernetes.io/projected/63ac6df1-998d-43e6-bf49-2aeff97e4a4b-kube-api-access-tlb5t\") pod \"cert-manager-webhook-5dbbbf75cc-c5hhg\" (UID: \"63ac6df1-998d-43e6-bf49-2aeff97e4a4b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:03.156177 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.156143 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63ac6df1-998d-43e6-bf49-2aeff97e4a4b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-c5hhg\" (UID: \"63ac6df1-998d-43e6-bf49-2aeff97e4a4b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:03.156177 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.156183 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlb5t\" (UniqueName: \"kubernetes.io/projected/63ac6df1-998d-43e6-bf49-2aeff97e4a4b-kube-api-access-tlb5t\") pod \"cert-manager-webhook-5dbbbf75cc-c5hhg\" (UID: \"63ac6df1-998d-43e6-bf49-2aeff97e4a4b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:03.203178 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.203151 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlb5t\" (UniqueName: \"kubernetes.io/projected/63ac6df1-998d-43e6-bf49-2aeff97e4a4b-kube-api-access-tlb5t\") pod \"cert-manager-webhook-5dbbbf75cc-c5hhg\" (UID: \"63ac6df1-998d-43e6-bf49-2aeff97e4a4b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:03.232961 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.232939 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63ac6df1-998d-43e6-bf49-2aeff97e4a4b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-c5hhg\" (UID: \"63ac6df1-998d-43e6-bf49-2aeff97e4a4b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:03.294433 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.294394 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" event={"ID":"8fe95351-fb90-49ed-8347-ac51bf4c03c5","Type":"ContainerStarted","Data":"94ea6dfd7554d678aba2dbf8d67faf2b8e3850b4caaf6fa7571ce0f6faf365b4"} Nov 03 23:33:03.296585 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.296555 6208 generic.go:356] "Generic (PLEG): container finished" podID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerID="94e37edde25ad9297af63bd2e0533c44fa1c73aa2745b3e1824240e69cb93221" exitCode=0 Nov 03 23:33:03.296735 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.296634 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" event={"ID":"1b6e8876-231d-4e24-a4cc-f60fb0db11d5","Type":"ContainerDied","Data":"94e37edde25ad9297af63bd2e0533c44fa1c73aa2745b3e1824240e69cb93221"} Nov 03 23:33:03.298852 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.298825 6208 generic.go:356] "Generic (PLEG): container finished" podID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerID="da2bdaa769eeeedd20b195e2f58a060e08b120e776e679b2e9f106f4e0ace07b" exitCode=0 Nov 03 23:33:03.298979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.298900 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" event={"ID":"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b","Type":"ContainerDied","Data":"da2bdaa769eeeedd20b195e2f58a060e08b120e776e679b2e9f106f4e0ace07b"} Nov 03 23:33:03.301041 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.301000 6208 generic.go:356] "Generic (PLEG): container finished" podID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerID="bdebc0ba2e828585637bf3f5f5511334f5b0df445aa4dfadededecd68904f3df" exitCode=0 Nov 03 23:33:03.301148 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.301069 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" event={"ID":"92e5c088-8670-4d9c-ad61-7e5ce4e55649","Type":"ContainerDied","Data":"bdebc0ba2e828585637bf3f5f5511334f5b0df445aa4dfadededecd68904f3df"} Nov 03 23:33:03.330522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.330430 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" podStartSLOduration=3.1109409279999998 podStartE2EDuration="35.330415483s" totalImagesPullingTime="32.219474555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:32:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:03.327949299 +0000 UTC m=+202.150545463" watchObservedRunningTime="2026-11-03 23:33:03.330415483 +0000 UTC m=+202.153011650" Nov 03 23:33:03.378004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.377971 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:03.412989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.412965 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:33:03.458458 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.458435 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-util\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-util\") pod \"90904130-60ec-4811-b110-4ed57f4ad6a7\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " Nov 03 23:33:03.458574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.458522 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-bundle\") pod \"90904130-60ec-4811-b110-4ed57f4ad6a7\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " Nov 03 23:33:03.458574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.458551 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90904130-60ec-4811-b110-4ed57f4ad6a7-kube-api-access-fc9v4\" (UniqueName: \"kubernetes.io/projected/90904130-60ec-4811-b110-4ed57f4ad6a7-kube-api-access-fc9v4\") pod \"90904130-60ec-4811-b110-4ed57f4ad6a7\" (UID: \"90904130-60ec-4811-b110-4ed57f4ad6a7\") " Nov 03 23:33:03.459456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.459412 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-bundle" pod "90904130-60ec-4811-b110-4ed57f4ad6a7" (UID: "90904130-60ec-4811-b110-4ed57f4ad6a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:03.461376 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.461106 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90904130-60ec-4811-b110-4ed57f4ad6a7-kube-api-access-fc9v4" pod "90904130-60ec-4811-b110-4ed57f4ad6a7" (UID: "90904130-60ec-4811-b110-4ed57f4ad6a7"). InnerVolumeSpecName "kube-api-access-fc9v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:03.463681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.463655 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-util" pod "90904130-60ec-4811-b110-4ed57f4ad6a7" (UID: "90904130-60ec-4811-b110-4ed57f4ad6a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:03.559114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.559085 6208 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-util\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:03.559114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.559109 6208 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90904130-60ec-4811-b110-4ed57f4ad6a7-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:03.559114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.559118 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc9v4\" (UniqueName: \"kubernetes.io/projected/90904130-60ec-4811-b110-4ed57f4ad6a7-kube-api-access-fc9v4\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:03.663806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:03.663731 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg"] Nov 03 23:33:03.667097 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:33:03.667069 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63ac6df1_998d_43e6_bf49_2aeff97e4a4b.slice/crio-64ce937cfe18a054c70140870d547894690684460c4c5160052fb4128a8cd976 WatchSource:0}: Error finding container 64ce937cfe18a054c70140870d547894690684460c4c5160052fb4128a8cd976: Status 404 returned error can't find the container with id 64ce937cfe18a054c70140870d547894690684460c4c5160052fb4128a8cd976 Nov 03 23:33:04.024522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.024480 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:04.305915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.305823 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" event={"ID":"63ac6df1-998d-43e6-bf49-2aeff97e4a4b","Type":"ContainerStarted","Data":"64ce937cfe18a054c70140870d547894690684460c4c5160052fb4128a8cd976"} Nov 03 23:33:04.307553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.307528 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" event={"ID":"90904130-60ec-4811-b110-4ed57f4ad6a7","Type":"ContainerDied","Data":"e39cbacec01d3edd31a0edc02a0a8dcd7403b91b319921c364fb51ea8195316d"} Nov 03 23:33:04.307553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.307551 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c29nf4" Nov 03 23:33:04.307709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.307571 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e39cbacec01d3edd31a0edc02a0a8dcd7403b91b319921c364fb51ea8195316d" Nov 03 23:33:04.309611 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.309590 6208 generic.go:356] "Generic (PLEG): container finished" podID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerID="94ea6dfd7554d678aba2dbf8d67faf2b8e3850b4caaf6fa7571ce0f6faf365b4" exitCode=0 Nov 03 23:33:04.309717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.309668 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" event={"ID":"8fe95351-fb90-49ed-8347-ac51bf4c03c5","Type":"ContainerDied","Data":"94ea6dfd7554d678aba2dbf8d67faf2b8e3850b4caaf6fa7571ce0f6faf365b4"} Nov 03 23:33:04.430441 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.430384 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:04.443759 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.443733 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:33:04.468148 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.468124 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-util\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-util\") pod \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " Nov 03 23:33:04.468275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.468224 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-kube-api-access-55r9g\" (UniqueName: \"kubernetes.io/projected/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-kube-api-access-55r9g\") pod \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " Nov 03 23:33:04.468275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.468260 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-bundle\") pod \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\" (UID: \"1b6e8876-231d-4e24-a4cc-f60fb0db11d5\") " Nov 03 23:33:04.469649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.469515 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-bundle" pod "1b6e8876-231d-4e24-a4cc-f60fb0db11d5" (UID: "1b6e8876-231d-4e24-a4cc-f60fb0db11d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:04.474265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.473165 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-kube-api-access-55r9g" pod "1b6e8876-231d-4e24-a4cc-f60fb0db11d5" (UID: "1b6e8876-231d-4e24-a4cc-f60fb0db11d5"). InnerVolumeSpecName "kube-api-access-55r9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:04.475866 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.475835 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-util" pod "1b6e8876-231d-4e24-a4cc-f60fb0db11d5" (UID: "1b6e8876-231d-4e24-a4cc-f60fb0db11d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:04.481668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.481648 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:33:04.486185 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.486165 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:33:04.569020 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.568946 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-bundle\") pod \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " Nov 03 23:33:04.569020 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.568988 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-util\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-util\") pod \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " Nov 03 23:33:04.569247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.569072 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e5c088-8670-4d9c-ad61-7e5ce4e55649-kube-api-access-zvhb4\" (UniqueName: \"kubernetes.io/projected/92e5c088-8670-4d9c-ad61-7e5ce4e55649-kube-api-access-zvhb4\") pod \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " Nov 03 23:33:04.569247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.569116 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-bundle\") pod \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " Nov 03 23:33:04.569247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.569146 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-kube-api-access-m5qm4\" (UniqueName: \"kubernetes.io/projected/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-kube-api-access-m5qm4\") pod \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\" (UID: \"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b\") " Nov 03 23:33:04.569247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.569186 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-util\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-util\") pod \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\" (UID: \"92e5c088-8670-4d9c-ad61-7e5ce4e55649\") " Nov 03 23:33:04.569455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.569388 6208 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-util\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:04.569455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.569407 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55r9g\" (UniqueName: \"kubernetes.io/projected/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-kube-api-access-55r9g\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:04.569455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.569424 6208 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b6e8876-231d-4e24-a4cc-f60fb0db11d5-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:04.569605 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.569586 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-bundle" pod "1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" (UID: "1e98d1d6-dc5d-4571-954e-5700f7cf2f7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:04.569785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.569755 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-bundle" pod "92e5c088-8670-4d9c-ad61-7e5ce4e55649" (UID: "92e5c088-8670-4d9c-ad61-7e5ce4e55649"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:04.571369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.571347 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-kube-api-access-m5qm4" pod "1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" (UID: "1e98d1d6-dc5d-4571-954e-5700f7cf2f7b"). InnerVolumeSpecName "kube-api-access-m5qm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:04.571536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.571519 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e5c088-8670-4d9c-ad61-7e5ce4e55649-kube-api-access-zvhb4" pod "92e5c088-8670-4d9c-ad61-7e5ce4e55649" (UID: "92e5c088-8670-4d9c-ad61-7e5ce4e55649"). InnerVolumeSpecName "kube-api-access-zvhb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:04.573883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.573863 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-util" pod "92e5c088-8670-4d9c-ad61-7e5ce4e55649" (UID: "92e5c088-8670-4d9c-ad61-7e5ce4e55649"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:04.575661 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.575639 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-util" pod "1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" (UID: "1e98d1d6-dc5d-4571-954e-5700f7cf2f7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:04.670458 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.670424 6208 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-util\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:04.670458 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.670452 6208 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:04.670458 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.670464 6208 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-util\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:04.670678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.670476 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhb4\" (UniqueName: \"kubernetes.io/projected/92e5c088-8670-4d9c-ad61-7e5ce4e55649-kube-api-access-zvhb4\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:04.670678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.670487 6208 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92e5c088-8670-4d9c-ad61-7e5ce4e55649-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:04.670678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.670501 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qm4\" (UniqueName: \"kubernetes.io/projected/1e98d1d6-dc5d-4571-954e-5700f7cf2f7b-kube-api-access-m5qm4\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:04.744529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744497 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps"] Nov 03 23:33:04.744760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744749 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerName="pull" Nov 03 23:33:04.744807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744762 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerName="pull" Nov 03 23:33:04.744807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744774 6208 cpu_manager.go:414] "containerMap: removing container" podUID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerName="pull" Nov 03 23:33:04.744807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744780 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerName="pull" Nov 03 23:33:04.744807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744788 6208 cpu_manager.go:414] "containerMap: removing container" podUID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerName="extract" Nov 03 23:33:04.744807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744793 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerName="extract" Nov 03 23:33:04.744807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744799 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerName="extract" Nov 03 23:33:04.744807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744805 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerName="extract" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744813 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerName="extract" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744818 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerName="extract" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744825 6208 cpu_manager.go:414] "containerMap: removing container" podUID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerName="util" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744829 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerName="util" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744841 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerName="util" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744845 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerName="util" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744852 6208 cpu_manager.go:414] "containerMap: removing container" podUID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerName="pull" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744856 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerName="pull" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744863 6208 cpu_manager.go:414] "containerMap: removing container" podUID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerName="extract" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744867 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerName="extract" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744876 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerName="util" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744880 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerName="util" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744886 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerName="pull" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744890 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerName="pull" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744899 6208 cpu_manager.go:414] "containerMap: removing container" podUID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerName="util" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744903 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerName="util" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744942 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6e8876-231d-4e24-a4cc-f60fb0db11d5" containerName="extract" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744951 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e5c088-8670-4d9c-ad61-7e5ce4e55649" containerName="extract" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744959 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="90904130-60ec-4811-b110-4ed57f4ad6a7" containerName="extract" Nov 03 23:33:04.744999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.744965 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e98d1d6-dc5d-4571-954e-5700f7cf2f7b" containerName="extract" Nov 03 23:33:04.746594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.746570 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" Nov 03 23:33:04.751655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.751626 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-g7gjq\"" Nov 03 23:33:04.771409 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.771372 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4knr\" (UniqueName: \"kubernetes.io/projected/06b6b9d6-7ac8-4541-936f-338efb90765f-kube-api-access-t4knr\") pod \"cert-manager-cainjector-5c57bbd4fc-58hps\" (UID: \"06b6b9d6-7ac8-4541-936f-338efb90765f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" Nov 03 23:33:04.771532 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.771448 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06b6b9d6-7ac8-4541-936f-338efb90765f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-58hps\" (UID: \"06b6b9d6-7ac8-4541-936f-338efb90765f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" Nov 03 23:33:04.772683 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.772645 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps"] Nov 03 23:33:04.872257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.872181 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4knr\" (UniqueName: \"kubernetes.io/projected/06b6b9d6-7ac8-4541-936f-338efb90765f-kube-api-access-t4knr\") pod \"cert-manager-cainjector-5c57bbd4fc-58hps\" (UID: \"06b6b9d6-7ac8-4541-936f-338efb90765f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" Nov 03 23:33:04.872257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.872239 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06b6b9d6-7ac8-4541-936f-338efb90765f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-58hps\" (UID: \"06b6b9d6-7ac8-4541-936f-338efb90765f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" Nov 03 23:33:04.886440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.886413 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06b6b9d6-7ac8-4541-936f-338efb90765f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-58hps\" (UID: \"06b6b9d6-7ac8-4541-936f-338efb90765f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" Nov 03 23:33:04.886763 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:04.886741 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4knr\" (UniqueName: \"kubernetes.io/projected/06b6b9d6-7ac8-4541-936f-338efb90765f-kube-api-access-t4knr\") pod \"cert-manager-cainjector-5c57bbd4fc-58hps\" (UID: \"06b6b9d6-7ac8-4541-936f-338efb90765f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" Nov 03 23:33:05.058163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.058131 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" Nov 03 23:33:05.315777 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.315757 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" Nov 03 23:33:05.316047 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.315784 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f88np" event={"ID":"1b6e8876-231d-4e24-a4cc-f60fb0db11d5","Type":"ContainerDied","Data":"9b7fcb7b1f749c10c216f00f50deda2e747fc6a4ff267f43bb82fb5baaa66989"} Nov 03 23:33:05.316047 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.316007 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b7fcb7b1f749c10c216f00f50deda2e747fc6a4ff267f43bb82fb5baaa66989" Nov 03 23:33:05.318395 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.318369 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" event={"ID":"1e98d1d6-dc5d-4571-954e-5700f7cf2f7b","Type":"ContainerDied","Data":"695b4c134ef5b3d2bd4d3b96edad5018eb38896ca63332e63e47f68c39238933"} Nov 03 23:33:05.318479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.318413 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="695b4c134ef5b3d2bd4d3b96edad5018eb38896ca63332e63e47f68c39238933" Nov 03 23:33:05.318479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.318424 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b894n" Nov 03 23:33:05.322023 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.321922 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" Nov 03 23:33:05.322273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.322246 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccjfsb" event={"ID":"92e5c088-8670-4d9c-ad61-7e5ce4e55649","Type":"ContainerDied","Data":"218b6b3c1693f9c4fba20c52ed3fc666eb05743e08df73bd9c74c8b699c9e5f3"} Nov 03 23:33:05.322384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.322290 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="218b6b3c1693f9c4fba20c52ed3fc666eb05743e08df73bd9c74c8b699c9e5f3" Nov 03 23:33:05.341380 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.341252 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps"] Nov 03 23:33:05.344224 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:33:05.344194 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06b6b9d6_7ac8_4541_936f_338efb90765f.slice/crio-35b3c719b012d6370d42268a1bff946e9adcba9b005f9f6a0a46540429f860de WatchSource:0}: Error finding container 35b3c719b012d6370d42268a1bff946e9adcba9b005f9f6a0a46540429f860de: Status 404 returned error can't find the container with id 35b3c719b012d6370d42268a1bff946e9adcba9b005f9f6a0a46540429f860de Nov 03 23:33:05.457696 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.457672 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:33:05.477314 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.477287 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-util\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-util\") pod \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " Nov 03 23:33:05.477463 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.477379 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-bundle\") pod \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " Nov 03 23:33:05.477463 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.477423 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe95351-fb90-49ed-8347-ac51bf4c03c5-kube-api-access-bkth8\" (UniqueName: \"kubernetes.io/projected/8fe95351-fb90-49ed-8347-ac51bf4c03c5-kube-api-access-bkth8\") pod \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\" (UID: \"8fe95351-fb90-49ed-8347-ac51bf4c03c5\") " Nov 03 23:33:05.478318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.478295 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-bundle" pod "8fe95351-fb90-49ed-8347-ac51bf4c03c5" (UID: "8fe95351-fb90-49ed-8347-ac51bf4c03c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:05.479491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.479468 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe95351-fb90-49ed-8347-ac51bf4c03c5-kube-api-access-bkth8" pod "8fe95351-fb90-49ed-8347-ac51bf4c03c5" (UID: "8fe95351-fb90-49ed-8347-ac51bf4c03c5"). InnerVolumeSpecName "kube-api-access-bkth8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:33:05.484941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.484915 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-util" pod "8fe95351-fb90-49ed-8347-ac51bf4c03c5" (UID: "8fe95351-fb90-49ed-8347-ac51bf4c03c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:33:05.578174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.578099 6208 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-util\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:05.578174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.578127 6208 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe95351-fb90-49ed-8347-ac51bf4c03c5-bundle\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:05.578174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:05.578165 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkth8\" (UniqueName: \"kubernetes.io/projected/8fe95351-fb90-49ed-8347-ac51bf4c03c5-kube-api-access-bkth8\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:33:05.705606 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:33:05.705572 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:33:06.328709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:06.328667 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" event={"ID":"06b6b9d6-7ac8-4541-936f-338efb90765f","Type":"ContainerStarted","Data":"35b3c719b012d6370d42268a1bff946e9adcba9b005f9f6a0a46540429f860de"} Nov 03 23:33:06.331137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:06.331105 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" event={"ID":"8fe95351-fb90-49ed-8347-ac51bf4c03c5","Type":"ContainerDied","Data":"e16d3a80469af59c4664c9d8fe8db3362fb2ad7a1d48b0f5c219931b18d97424"} Nov 03 23:33:06.331137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:06.331131 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azzb5s" Nov 03 23:33:06.331330 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:06.331160 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e16d3a80469af59c4664c9d8fe8db3362fb2ad7a1d48b0f5c219931b18d97424" Nov 03 23:33:07.337240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:07.337207 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" event={"ID":"63ac6df1-998d-43e6-bf49-2aeff97e4a4b","Type":"ContainerStarted","Data":"f317f2633735528306440b17a85c3e6979df2952ef0a98230ff91cca400a421e"} Nov 03 23:33:07.337706 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:07.337331 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:07.360692 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:07.360640 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" podStartSLOduration=2.209750558 podStartE2EDuration="5.360627699s" totalImagesPullingTime="3.150877141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:07.360307552 +0000 UTC m=+206.182903714" watchObservedRunningTime="2026-11-03 23:33:07.360627699 +0000 UTC m=+206.183223861" Nov 03 23:33:08.343631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:08.343581 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" event={"ID":"06b6b9d6-7ac8-4541-936f-338efb90765f","Type":"ContainerStarted","Data":"e68aea5710ca9ae92d8ddfc8374e365125fed8ddf1d3ff078793103aa1ab6ec9"} Nov 03 23:33:08.365474 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:08.365422 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-58hps" podStartSLOduration=1.781667677 podStartE2EDuration="4.365405602s" totalImagesPullingTime="2.583737925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:08.363403834 +0000 UTC m=+207.185999999" watchObservedRunningTime="2026-11-03 23:33:08.365405602 +0000 UTC m=+207.188001758" Nov 03 23:33:09.992806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:09.992768 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:10.105259 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:10.105216 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:11.877103 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.877061 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:11.939587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.937227 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn"] Nov 03 23:33:11.939587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.937870 6208 cpu_manager.go:414] "containerMap: removing container" podUID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerName="extract" Nov 03 23:33:11.939587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.937886 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerName="extract" Nov 03 23:33:11.939587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.937906 6208 cpu_manager.go:414] "containerMap: removing container" podUID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerName="pull" Nov 03 23:33:11.939587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.937914 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerName="pull" Nov 03 23:33:11.939587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.937960 6208 cpu_manager.go:414] "containerMap: removing container" podUID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerName="util" Nov 03 23:33:11.939587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.937968 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerName="util" Nov 03 23:33:11.939587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.938115 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe95351-fb90-49ed-8347-ac51bf4c03c5" containerName="extract" Nov 03 23:33:11.941888 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.941859 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" Nov 03 23:33:11.950932 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.950910 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn"] Nov 03 23:33:11.953715 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.953688 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 23:33:11.953798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.953746 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 23:33:11.953798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:11.953771 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-894rm\"" Nov 03 23:33:12.030929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:12.030899 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50e5736e-cc05-4ce1-801b-570844a9a436-tmp\") pod \"jobset-operator-674cd59dc5-nngcn\" (UID: \"50e5736e-cc05-4ce1-801b-570844a9a436\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" Nov 03 23:33:12.031092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:12.030951 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mk5\" (UniqueName: \"kubernetes.io/projected/50e5736e-cc05-4ce1-801b-570844a9a436-kube-api-access-m2mk5\") pod \"jobset-operator-674cd59dc5-nngcn\" (UID: \"50e5736e-cc05-4ce1-801b-570844a9a436\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" Nov 03 23:33:12.132121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:12.132018 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50e5736e-cc05-4ce1-801b-570844a9a436-tmp\") pod \"jobset-operator-674cd59dc5-nngcn\" (UID: \"50e5736e-cc05-4ce1-801b-570844a9a436\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" Nov 03 23:33:12.132272 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:12.132128 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mk5\" (UniqueName: \"kubernetes.io/projected/50e5736e-cc05-4ce1-801b-570844a9a436-kube-api-access-m2mk5\") pod \"jobset-operator-674cd59dc5-nngcn\" (UID: \"50e5736e-cc05-4ce1-801b-570844a9a436\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" Nov 03 23:33:12.132386 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:12.132361 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50e5736e-cc05-4ce1-801b-570844a9a436-tmp\") pod \"jobset-operator-674cd59dc5-nngcn\" (UID: \"50e5736e-cc05-4ce1-801b-570844a9a436\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" Nov 03 23:33:12.147714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:12.147673 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mk5\" (UniqueName: \"kubernetes.io/projected/50e5736e-cc05-4ce1-801b-570844a9a436-kube-api-access-m2mk5\") pod \"jobset-operator-674cd59dc5-nngcn\" (UID: \"50e5736e-cc05-4ce1-801b-570844a9a436\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" Nov 03 23:33:12.253879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:12.253842 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" Nov 03 23:33:12.529819 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:12.529673 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn"] Nov 03 23:33:12.532478 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:33:12.532455 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50e5736e_cc05_4ce1_801b_570844a9a436.slice/crio-8f91f58c9e2d5d897dbf51ce20541d0c372234df1b8f72c66c8438369dc4bc38 WatchSource:0}: Error finding container 8f91f58c9e2d5d897dbf51ce20541d0c372234df1b8f72c66c8438369dc4bc38: Status 404 returned error can't find the container with id 8f91f58c9e2d5d897dbf51ce20541d0c372234df1b8f72c66c8438369dc4bc38 Nov 03 23:33:13.345832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.345805 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" Nov 03 23:33:13.366400 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.366357 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" event={"ID":"50e5736e-cc05-4ce1-801b-570844a9a436","Type":"ContainerStarted","Data":"8f91f58c9e2d5d897dbf51ce20541d0c372234df1b8f72c66c8438369dc4bc38"} Nov 03 23:33:13.530610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.530583 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-mcznz"] Nov 03 23:33:13.534753 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.534736 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mcznz" Nov 03 23:33:13.545246 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.545224 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n96w8\"" Nov 03 23:33:13.546479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.546460 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mcznz"] Nov 03 23:33:13.643760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.643691 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b1afcd-8bf2-49ff-b50d-5cccb24ae831-bound-sa-token\") pod \"cert-manager-597b95cb84-mcznz\" (UID: \"42b1afcd-8bf2-49ff-b50d-5cccb24ae831\") " pod="cert-manager/cert-manager-597b95cb84-mcznz" Nov 03 23:33:13.643760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.643736 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5vq\" (UniqueName: \"kubernetes.io/projected/42b1afcd-8bf2-49ff-b50d-5cccb24ae831-kube-api-access-zw5vq\") pod \"cert-manager-597b95cb84-mcznz\" (UID: \"42b1afcd-8bf2-49ff-b50d-5cccb24ae831\") " pod="cert-manager/cert-manager-597b95cb84-mcznz" Nov 03 23:33:13.744468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.744438 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5vq\" (UniqueName: \"kubernetes.io/projected/42b1afcd-8bf2-49ff-b50d-5cccb24ae831-kube-api-access-zw5vq\") pod \"cert-manager-597b95cb84-mcznz\" (UID: \"42b1afcd-8bf2-49ff-b50d-5cccb24ae831\") " pod="cert-manager/cert-manager-597b95cb84-mcznz" Nov 03 23:33:13.744616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.744504 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b1afcd-8bf2-49ff-b50d-5cccb24ae831-bound-sa-token\") pod \"cert-manager-597b95cb84-mcznz\" (UID: \"42b1afcd-8bf2-49ff-b50d-5cccb24ae831\") " pod="cert-manager/cert-manager-597b95cb84-mcznz" Nov 03 23:33:13.757356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.757322 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42b1afcd-8bf2-49ff-b50d-5cccb24ae831-bound-sa-token\") pod \"cert-manager-597b95cb84-mcznz\" (UID: \"42b1afcd-8bf2-49ff-b50d-5cccb24ae831\") " pod="cert-manager/cert-manager-597b95cb84-mcznz" Nov 03 23:33:13.757473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.757454 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5vq\" (UniqueName: \"kubernetes.io/projected/42b1afcd-8bf2-49ff-b50d-5cccb24ae831-kube-api-access-zw5vq\") pod \"cert-manager-597b95cb84-mcznz\" (UID: \"42b1afcd-8bf2-49ff-b50d-5cccb24ae831\") " pod="cert-manager/cert-manager-597b95cb84-mcznz" Nov 03 23:33:13.846177 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:13.846145 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mcznz" Nov 03 23:33:14.128332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:14.128099 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mcznz"] Nov 03 23:33:14.135185 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:33:14.135156 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42b1afcd_8bf2_49ff_b50d_5cccb24ae831.slice/crio-265e5c7f8f2b2bd3b2c33f1257f1022e5635117bbc9873a5e49bb8722b0034a8 WatchSource:0}: Error finding container 265e5c7f8f2b2bd3b2c33f1257f1022e5635117bbc9873a5e49bb8722b0034a8: Status 404 returned error can't find the container with id 265e5c7f8f2b2bd3b2c33f1257f1022e5635117bbc9873a5e49bb8722b0034a8 Nov 03 23:33:14.373391 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:14.373351 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mcznz" event={"ID":"42b1afcd-8bf2-49ff-b50d-5cccb24ae831","Type":"ContainerStarted","Data":"dfa089bbad43cf42328eb304178986818f478cdb15e693c84aedfe4f31e5eb6c"} Nov 03 23:33:14.373391 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:14.373397 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mcznz" event={"ID":"42b1afcd-8bf2-49ff-b50d-5cccb24ae831","Type":"ContainerStarted","Data":"265e5c7f8f2b2bd3b2c33f1257f1022e5635117bbc9873a5e49bb8722b0034a8"} Nov 03 23:33:14.393847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:14.393792 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-mcznz" podStartSLOduration=1.393778981 podStartE2EDuration="1.393778981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:14.392296836 +0000 UTC m=+213.214892997" watchObservedRunningTime="2026-11-03 23:33:14.393778981 +0000 UTC m=+213.216375142" Nov 03 23:33:16.384000 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:16.383962 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" event={"ID":"50e5736e-cc05-4ce1-801b-570844a9a436","Type":"ContainerStarted","Data":"f417fcd6a3ba71168557436d50712366a4f6d9587ad40a83d8159fed6087e0fb"} Nov 03 23:33:16.414935 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:16.414881 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nngcn" podStartSLOduration=2.311351385 podStartE2EDuration="5.41486706s" totalImagesPullingTime="3.103515675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:16.412727146 +0000 UTC m=+215.235323320" watchObservedRunningTime="2026-11-03 23:33:16.41486706 +0000 UTC m=+215.237463221" Nov 03 23:33:18.127261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:18.127227 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:19.028928 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:19.028896 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:21.140747 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:21.140708 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:21.246581 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:21.246544 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:22.579973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.579938 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:22.804782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.804747 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7"] Nov 03 23:33:22.808020 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.807997 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:22.842915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.842838 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7"] Nov 03 23:33:22.870971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.870944 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 23:33:22.871306 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.871279 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 23:33:22.872131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.872117 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 23:33:22.872256 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.872244 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bc7w7\"" Nov 03 23:33:22.872324 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.872270 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 23:33:22.872324 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.872297 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 23:33:22.918409 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.918370 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-webhook-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:22.918666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.918645 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-manifests\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:22.918776 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.918681 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82mnk\" (UniqueName: \"kubernetes.io/projected/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-kube-api-access-82mnk\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:22.918776 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.918714 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-metrics-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:22.918865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.918786 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-tmp\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:22.918865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.918845 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:22.983799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:22.983762 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:23.020297 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.020263 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82mnk\" (UniqueName: \"kubernetes.io/projected/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-kube-api-access-82mnk\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.020457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.020306 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-metrics-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.020457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.020331 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-tmp\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.020457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.020374 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.020457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.020430 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-webhook-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.020628 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.020471 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-manifests\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.020868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.020849 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-manifests\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.022950 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.022923 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-tmp\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.023081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.022978 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-webhook-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.023081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.023022 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-metrics-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.023199 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.023109 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.040669 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.040644 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82mnk\" (UniqueName: \"kubernetes.io/projected/e27fd8a4-1210-4b3a-8787-63ac8da2a2ad-kube-api-access-82mnk\") pod \"opendatahub-operator-controller-manager-775bfbdf99-qc9b7\" (UID: \"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.105817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.105726 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:23.126690 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.126663 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:23.217561 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.217509 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:23.433226 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.433146 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:23.439982 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:23.439954 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7"] Nov 03 23:33:23.444931 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:33:23.444905 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27fd8a4_1210_4b3a_8787_63ac8da2a2ad.slice/crio-1f9de5ba5ec168945bda060731f71fbd8193f4f3f1315a742597b00d477e0395 WatchSource:0}: Error finding container 1f9de5ba5ec168945bda060731f71fbd8193f4f3f1315a742597b00d477e0395: Status 404 returned error can't find the container with id 1f9de5ba5ec168945bda060731f71fbd8193f4f3f1315a742597b00d477e0395 Nov 03 23:33:24.419761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:24.419727 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" event={"ID":"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad","Type":"ContainerStarted","Data":"1f9de5ba5ec168945bda060731f71fbd8193f4f3f1315a742597b00d477e0395"} Nov 03 23:33:24.530079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:24.530027 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:24.680996 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:24.680919 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:25.066358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:25.066322 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:25.212424 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:25.212377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:25.490198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:25.490163 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:26.038377 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:26.038340 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:26.708716 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:26.708681 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:26.874952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:26.874916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:27.436275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:27.436188 6208 generic.go:356] "Generic (PLEG): container finished" podID="e27fd8a4-1210-4b3a-8787-63ac8da2a2ad" containerID="0a645a0a9dfb5811ddbdc2588a36421abc4475a5f8739070584f80a10dd618d3" exitCode=0 Nov 03 23:33:27.436275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:27.436260 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" event={"ID":"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad","Type":"ContainerDied","Data":"0a645a0a9dfb5811ddbdc2588a36421abc4475a5f8739070584f80a10dd618d3"} Nov 03 23:33:28.442178 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:28.442099 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" event={"ID":"e27fd8a4-1210-4b3a-8787-63ac8da2a2ad","Type":"ContainerStarted","Data":"f6e727541fe55183e8bd33c0c187fb3d050e62e858d19d2e2891e77f99641923"} Nov 03 23:33:28.442558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:28.442235 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:28.501629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:28.501579 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" podStartSLOduration=2.3963937299999998 podStartE2EDuration="6.501566632s" totalImagesPullingTime="3.531057261s" totalInitContainerRuntime="574.115641ms" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:28.49971094 +0000 UTC m=+227.322307113" watchObservedRunningTime="2026-11-03 23:33:28.501566632 +0000 UTC m=+227.324162793" Nov 03 23:33:28.757404 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:28.757364 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:32.121079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:32.121046 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:32.353720 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:32.353686 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:32.493680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:32.493650 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:32.724187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:32.724145 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:35.702331 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:33:35.702292 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:33:35.908085 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:35.908051 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:38.458305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.458269 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k"] Nov 03 23:33:38.462679 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.462660 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.472263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.472239 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k"] Nov 03 23:33:38.474067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.474014 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 23:33:38.474220 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.474083 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 23:33:38.474220 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.474095 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 23:33:38.474220 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.474095 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 23:33:38.474220 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.474190 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pkmb4\"" Nov 03 23:33:38.496829 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.496778 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:38.553010 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.552975 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmxs\" (UniqueName: \"kubernetes.io/projected/fd69111b-a7ff-4661-8608-4dbc46282b48-kube-api-access-ncmxs\") pod \"kuadrant-console-plugin-66c4fb575d-jc64k\" (UID: \"fd69111b-a7ff-4661-8608-4dbc46282b48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.553195 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.553012 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fd69111b-a7ff-4661-8608-4dbc46282b48-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jc64k\" (UID: \"fd69111b-a7ff-4661-8608-4dbc46282b48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.553195 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.553151 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd69111b-a7ff-4661-8608-4dbc46282b48-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jc64k\" (UID: \"fd69111b-a7ff-4661-8608-4dbc46282b48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.653537 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.653502 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd69111b-a7ff-4661-8608-4dbc46282b48-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jc64k\" (UID: \"fd69111b-a7ff-4661-8608-4dbc46282b48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.653676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.653577 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmxs\" (UniqueName: \"kubernetes.io/projected/fd69111b-a7ff-4661-8608-4dbc46282b48-kube-api-access-ncmxs\") pod \"kuadrant-console-plugin-66c4fb575d-jc64k\" (UID: \"fd69111b-a7ff-4661-8608-4dbc46282b48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.653676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.653603 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fd69111b-a7ff-4661-8608-4dbc46282b48-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jc64k\" (UID: \"fd69111b-a7ff-4661-8608-4dbc46282b48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.654305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.654284 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fd69111b-a7ff-4661-8608-4dbc46282b48-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jc64k\" (UID: \"fd69111b-a7ff-4661-8608-4dbc46282b48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.656011 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.655984 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd69111b-a7ff-4661-8608-4dbc46282b48-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jc64k\" (UID: \"fd69111b-a7ff-4661-8608-4dbc46282b48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.669778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.669753 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmxs\" (UniqueName: \"kubernetes.io/projected/fd69111b-a7ff-4661-8608-4dbc46282b48-kube-api-access-ncmxs\") pod \"kuadrant-console-plugin-66c4fb575d-jc64k\" (UID: \"fd69111b-a7ff-4661-8608-4dbc46282b48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:38.676156 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.676130 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:38.773494 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:38.773414 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" Nov 03 23:33:39.052417 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:39.052345 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k"] Nov 03 23:33:39.055464 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:33:39.055438 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd69111b_a7ff_4661_8608_4dbc46282b48.slice/crio-cfafac7b87074e02d0cf6dbcfb4053d26c9ef29b3eecf4b847ede10abd0067cb WatchSource:0}: Error finding container cfafac7b87074e02d0cf6dbcfb4053d26c9ef29b3eecf4b847ede10abd0067cb: Status 404 returned error can't find the container with id cfafac7b87074e02d0cf6dbcfb4053d26c9ef29b3eecf4b847ede10abd0067cb Nov 03 23:33:39.449090 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:39.448987 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-775bfbdf99-qc9b7" Nov 03 23:33:39.489159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:39.489131 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" event={"ID":"fd69111b-a7ff-4661-8608-4dbc46282b48","Type":"ContainerStarted","Data":"cfafac7b87074e02d0cf6dbcfb4053d26c9ef29b3eecf4b847ede10abd0067cb"} Nov 03 23:33:39.839935 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:39.839900 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:39.984329 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:39.984295 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:40.507891 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:40.507846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:40.634861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:40.634829 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:44.515654 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:44.515619 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" event={"ID":"fd69111b-a7ff-4661-8608-4dbc46282b48","Type":"ContainerStarted","Data":"57c15725124a9a6baccc1bd6375d7982f14f8cdcfad3744183b814e1454b3305"} Nov 03 23:33:44.536679 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:44.536629 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jc64k" podStartSLOduration=1.216860154 podStartE2EDuration="6.536615769s" totalImagesPullingTime="5.319755615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:33:44.53495552 +0000 UTC m=+243.357551683" watchObservedRunningTime="2026-11-03 23:33:44.536615769 +0000 UTC m=+243.359211930" Nov 03 23:33:44.861211 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:44.861178 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:46.030116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:46.030084 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:46.130496 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:46.130462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:46.693758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:46.693719 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:46.813007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:46.812957 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:47.248672 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:47.248635 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:47.349017 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:47.348973 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:53.583892 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:53.583861 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:54.469300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:54.469271 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:54.594057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:54.594007 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:54.882954 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:54.882850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:56.946487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:56.946436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:57.058617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.058581 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:57.183525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.183493 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:57.405861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.405829 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx"] Nov 03 23:33:57.409208 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.409188 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.423002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.422977 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 23:33:57.423184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.423106 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 23:33:57.424077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.424052 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx"] Nov 03 23:33:57.424357 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.424325 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 23:33:57.424462 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.424358 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 23:33:57.424525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.424507 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pc7rt\"" Nov 03 23:33:57.444144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.444112 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:57.516782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.516742 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4h2q\" (UniqueName: \"kubernetes.io/projected/ed7ca6de-f794-4ea4-9a65-79dbcde9d373-kube-api-access-s4h2q\") pod \"kube-auth-proxy-694bd9f98-rl6lx\" (UID: \"ed7ca6de-f794-4ea4-9a65-79dbcde9d373\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.516782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.516784 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7ca6de-f794-4ea4-9a65-79dbcde9d373-tls-certs\") pod \"kube-auth-proxy-694bd9f98-rl6lx\" (UID: \"ed7ca6de-f794-4ea4-9a65-79dbcde9d373\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.516989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.516808 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed7ca6de-f794-4ea4-9a65-79dbcde9d373-tmp\") pod \"kube-auth-proxy-694bd9f98-rl6lx\" (UID: \"ed7ca6de-f794-4ea4-9a65-79dbcde9d373\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.588878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.588844 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:57.618376 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.618347 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4h2q\" (UniqueName: \"kubernetes.io/projected/ed7ca6de-f794-4ea4-9a65-79dbcde9d373-kube-api-access-s4h2q\") pod \"kube-auth-proxy-694bd9f98-rl6lx\" (UID: \"ed7ca6de-f794-4ea4-9a65-79dbcde9d373\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.618549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.618389 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7ca6de-f794-4ea4-9a65-79dbcde9d373-tls-certs\") pod \"kube-auth-proxy-694bd9f98-rl6lx\" (UID: \"ed7ca6de-f794-4ea4-9a65-79dbcde9d373\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.618549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.618413 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed7ca6de-f794-4ea4-9a65-79dbcde9d373-tmp\") pod \"kube-auth-proxy-694bd9f98-rl6lx\" (UID: \"ed7ca6de-f794-4ea4-9a65-79dbcde9d373\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.620631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.620607 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed7ca6de-f794-4ea4-9a65-79dbcde9d373-tmp\") pod \"kube-auth-proxy-694bd9f98-rl6lx\" (UID: \"ed7ca6de-f794-4ea4-9a65-79dbcde9d373\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.620774 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.620757 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7ca6de-f794-4ea4-9a65-79dbcde9d373-tls-certs\") pod \"kube-auth-proxy-694bd9f98-rl6lx\" (UID: \"ed7ca6de-f794-4ea4-9a65-79dbcde9d373\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.642137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.642110 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4h2q\" (UniqueName: \"kubernetes.io/projected/ed7ca6de-f794-4ea4-9a65-79dbcde9d373-kube-api-access-s4h2q\") pod \"kube-auth-proxy-694bd9f98-rl6lx\" (UID: \"ed7ca6de-f794-4ea4-9a65-79dbcde9d373\") " pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.721485 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.721460 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" Nov 03 23:33:57.743233 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.743192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:57.933028 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:57.932561 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:58.001383 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:58.001359 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx"] Nov 03 23:33:58.003675 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:33:58.003650 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7ca6de_f794_4ea4_9a65_79dbcde9d373.slice/crio-e8fe029c874714c07210eb2d4a5270a57f1780ef17575c16a518cc56425830a4 WatchSource:0}: Error finding container e8fe029c874714c07210eb2d4a5270a57f1780ef17575c16a518cc56425830a4: Status 404 returned error can't find the container with id e8fe029c874714c07210eb2d4a5270a57f1780ef17575c16a518cc56425830a4 Nov 03 23:33:58.514592 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:58.514556 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:58.578511 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:58.578470 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" event={"ID":"ed7ca6de-f794-4ea4-9a65-79dbcde9d373","Type":"ContainerStarted","Data":"e8fe029c874714c07210eb2d4a5270a57f1780ef17575c16a518cc56425830a4"} Nov 03 23:33:58.684938 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:58.684886 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:58.797399 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:58.797218 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:59.054356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:59.054270 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:59.291062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:59.291000 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:59.510051 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:59.509988 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:59.625231 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:59.625197 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:33:59.961806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:59.961774 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6r6m6"] Nov 03 23:33:59.969663 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:59.969640 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" Nov 03 23:33:59.976010 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:59.975976 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jjxq7\"" Nov 03 23:33:59.976341 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:33:59.976319 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6r6m6"] Nov 03 23:34:00.042488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.042436 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h9b7\" (UniqueName: \"kubernetes.io/projected/e27123f6-c921-4f59-afaf-57e64e15bfd6-kube-api-access-6h9b7\") pod \"authorino-56c6c9ddd5-6r6m6\" (UID: \"e27123f6-c921-4f59-afaf-57e64e15bfd6\") " pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" Nov 03 23:34:00.107933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.107893 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7"] Nov 03 23:34:00.111647 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.111619 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.115969 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.115927 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 23:34:00.122295 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.122265 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7"] Nov 03 23:34:00.142933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.142902 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f6fefb-f056-4495-a778-187e86acc017-metrics-certs\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.143133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.142967 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/07f6fefb-f056-4495-a778-187e86acc017-manager-config\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.143133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.143088 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07f6fefb-f056-4495-a778-187e86acc017-cert\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.143260 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.143137 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h9b7\" (UniqueName: \"kubernetes.io/projected/e27123f6-c921-4f59-afaf-57e64e15bfd6-kube-api-access-6h9b7\") pod \"authorino-56c6c9ddd5-6r6m6\" (UID: \"e27123f6-c921-4f59-afaf-57e64e15bfd6\") " pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" Nov 03 23:34:00.143260 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.143206 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjs8\" (UniqueName: \"kubernetes.io/projected/07f6fefb-f056-4495-a778-187e86acc017-kube-api-access-pqjs8\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.158708 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.158672 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h9b7\" (UniqueName: \"kubernetes.io/projected/e27123f6-c921-4f59-afaf-57e64e15bfd6-kube-api-access-6h9b7\") pod \"authorino-56c6c9ddd5-6r6m6\" (UID: \"e27123f6-c921-4f59-afaf-57e64e15bfd6\") " pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" Nov 03 23:34:00.175919 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.175888 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 23:34:00.175919 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.175918 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 23:34:00.176187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.175891 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-sbhb6\"" Nov 03 23:34:00.244631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.244542 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjs8\" (UniqueName: \"kubernetes.io/projected/07f6fefb-f056-4495-a778-187e86acc017-kube-api-access-pqjs8\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.244631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.244615 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f6fefb-f056-4495-a778-187e86acc017-metrics-certs\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.244864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.244672 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/07f6fefb-f056-4495-a778-187e86acc017-manager-config\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.244864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.244711 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07f6fefb-f056-4495-a778-187e86acc017-cert\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.245698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.245645 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/07f6fefb-f056-4495-a778-187e86acc017-manager-config\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.250772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.250737 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f6fefb-f056-4495-a778-187e86acc017-metrics-certs\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.253500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.253454 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07f6fefb-f056-4495-a778-187e86acc017-cert\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.265825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.265794 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjs8\" (UniqueName: \"kubernetes.io/projected/07f6fefb-f056-4495-a778-187e86acc017-kube-api-access-pqjs8\") pod \"jobset-controller-manager-74c78cbdbf-gxkf7\" (UID: \"07f6fefb-f056-4495-a778-187e86acc017\") " pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.282506 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.282478 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" Nov 03 23:34:00.303541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.303509 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:00.427213 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.427193 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:00.563973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.563896 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6r6m6"] Nov 03 23:34:00.566824 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:34:00.566791 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode27123f6_c921_4f59_afaf_57e64e15bfd6.slice/crio-da5e38e332d63e4dc62ca8fcd5676aebb61e7d80dcc7482104e8feaad507e57c WatchSource:0}: Error finding container da5e38e332d63e4dc62ca8fcd5676aebb61e7d80dcc7482104e8feaad507e57c: Status 404 returned error can't find the container with id da5e38e332d63e4dc62ca8fcd5676aebb61e7d80dcc7482104e8feaad507e57c Nov 03 23:34:00.589111 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.589074 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" event={"ID":"ed7ca6de-f794-4ea4-9a65-79dbcde9d373","Type":"ContainerStarted","Data":"c8707bc6adc03601034cb3d9b8baf07214516c278c969d1e343c9b49f3ef4098"} Nov 03 23:34:00.590285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.590261 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" event={"ID":"e27123f6-c921-4f59-afaf-57e64e15bfd6","Type":"ContainerStarted","Data":"da5e38e332d63e4dc62ca8fcd5676aebb61e7d80dcc7482104e8feaad507e57c"} Nov 03 23:34:00.607945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.607900 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-694bd9f98-rl6lx" podStartSLOduration=1.212920955 podStartE2EDuration="3.607885922s" totalImagesPullingTime="2.394964967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:00.60713724 +0000 UTC m=+259.429733415" watchObservedRunningTime="2026-11-03 23:34:00.607885922 +0000 UTC m=+259.430482084" Nov 03 23:34:00.707244 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:00.707120 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7"] Nov 03 23:34:00.709366 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:34:00.709340 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f6fefb_f056_4495_a778_187e86acc017.slice/crio-3c40fc5901d45f545d77e116424b866c17cde70dbfda97ee94392e1ef28e9749 WatchSource:0}: Error finding container 3c40fc5901d45f545d77e116424b866c17cde70dbfda97ee94392e1ef28e9749: Status 404 returned error can't find the container with id 3c40fc5901d45f545d77e116424b866c17cde70dbfda97ee94392e1ef28e9749 Nov 03 23:34:01.598863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:01.598827 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" event={"ID":"07f6fefb-f056-4495-a778-187e86acc017","Type":"ContainerStarted","Data":"3c40fc5901d45f545d77e116424b866c17cde70dbfda97ee94392e1ef28e9749"} Nov 03 23:34:01.875144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:01.875105 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:01.978319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:01.978287 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:02.081870 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:02.081831 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:04.616691 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:04.616650 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" event={"ID":"07f6fefb-f056-4495-a778-187e86acc017","Type":"ContainerStarted","Data":"d329ea3624a5804bc05aa795197fd6cd78f857707c53b989a06120a768ffe716"} Nov 03 23:34:04.617156 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:04.616749 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:04.637432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:04.637388 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" podStartSLOduration=1.5634685799999999 podStartE2EDuration="4.637374634s" totalImagesPullingTime="3.073906054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:04.636929087 +0000 UTC m=+263.459525248" watchObservedRunningTime="2026-11-03 23:34:04.637374634 +0000 UTC m=+263.459970793" Nov 03 23:34:05.709932 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:34:05.709885 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:34:06.630335 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:06.630293 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" event={"ID":"e27123f6-c921-4f59-afaf-57e64e15bfd6","Type":"ContainerStarted","Data":"f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c"} Nov 03 23:34:06.650049 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:06.649977 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" podStartSLOduration=2.414452177 podStartE2EDuration="7.649963489s" totalImagesPullingTime="5.235511312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:34:06.648887557 +0000 UTC m=+265.471483737" watchObservedRunningTime="2026-11-03 23:34:06.649963489 +0000 UTC m=+265.472559651" Nov 03 23:34:07.445669 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:07.445631 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:07.582138 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:07.582094 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:07.687749 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:07.687720 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:07.835940 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:07.835909 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:07.946478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:07.946438 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:13.755135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:13.755090 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:13.855252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:13.855218 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:15.630626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:15.630593 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 03 23:34:18.620108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:18.620076 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:18.715854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:18.715824 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:21.491558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:21.491474 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:21.591846 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:21.591812 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:22.503287 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:22.503252 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:22.686351 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:22.686318 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:24.886310 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:24.886275 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:24.997024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:24.996988 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:25.676525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:25.676490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:27.318202 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:27.318164 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:27.421579 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:27.421532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:27.727217 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:27.727189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:27.828222 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:27.828180 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:28.106206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:28.106123 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:28.212878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:28.212837 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:31.122610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:31.122572 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:31.244688 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:31.244638 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:33.574665 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:33.574628 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:33.674841 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:33.674806 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:35.702145 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:34:35.702110 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:34:38.315191 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:38.315159 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:38.421133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:38.421086 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:41.311732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:41.311693 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:34:41.314142 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:41.314119 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:34:44.938360 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:44.938318 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:46.027149 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:46.027112 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:46.230375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:46.230337 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:46.336976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:46.336901 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:46.445473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:46.445433 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:49.734195 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:49.734148 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:49.952257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:49.952225 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:50.064912 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:50.064821 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:51.153498 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:51.153459 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:51.393558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:51.393511 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:55.407490 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:55.407458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:55.509060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:55.509009 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:55.641764 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:55.641734 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:55.741496 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:55.741465 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:55.848962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:55.848922 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:56.976359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:56.976301 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:57.159278 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:57.159225 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:34:57.258259 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:34:57.258175 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:05.702392 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:35:05.702350 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:35:09.427929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:09.427897 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:09.526732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:09.526700 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:29.543786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:29.543741 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:29.660985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:29.660949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:30.885950 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:30.885917 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:30.962929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:30.962894 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7"] Nov 03 23:35:30.966285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:30.966260 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:30.976121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:30.976099 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:35:30.976121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:30.976098 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:35:30.976291 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:30.976152 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:35:30.977410 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:30.977394 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 23:35:30.981141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:30.981118 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7"] Nov 03 23:35:31.002522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.002489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:31.012602 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.012574 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-metrics-certs\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.012737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.012621 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-maas-api-tls\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.012737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.012705 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92wmm\" (UniqueName: \"kubernetes.io/projected/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-kube-api-access-92wmm\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.012822 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.012759 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-openshift-service-ca\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.105477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.105438 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:31.113552 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.113525 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-openshift-service-ca\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.113709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.113569 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-metrics-certs\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.113709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.113619 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-maas-api-tls\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.113709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.113671 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92wmm\" (UniqueName: \"kubernetes.io/projected/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-kube-api-access-92wmm\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.114328 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.114304 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-openshift-service-ca\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.115992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.115963 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-metrics-certs\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.116100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.116084 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-maas-api-tls\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.125674 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.125654 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92wmm\" (UniqueName: \"kubernetes.io/projected/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-kube-api-access-92wmm\") pod \"maas-api-56cc74d699-9b9p7\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.279673 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.279629 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:31.583265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:31.583190 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7"] Nov 03 23:35:31.586127 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:35:31.586098 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ad1919_14dd_4a50_a2cd_b55d8e9dddf8.slice/crio-2c4569487ec229b4ed398d17b8f19b4fa7899f287b304b335c3b81d953c0e741 WatchSource:0}: Error finding container 2c4569487ec229b4ed398d17b8f19b4fa7899f287b304b335c3b81d953c0e741: Status 404 returned error can't find the container with id 2c4569487ec229b4ed398d17b8f19b4fa7899f287b304b335c3b81d953c0e741 Nov 03 23:35:32.009881 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:32.009841 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" event={"ID":"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8","Type":"ContainerStarted","Data":"2c4569487ec229b4ed398d17b8f19b4fa7899f287b304b335c3b81d953c0e741"} Nov 03 23:35:35.706349 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:35:35.706283 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:35:37.035358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:37.035324 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" event={"ID":"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8","Type":"ContainerStarted","Data":"1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424"} Nov 03 23:35:37.035777 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:37.035404 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:37.056984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:37.056936 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" podStartSLOduration=2.619182527 podStartE2EDuration="7.056923514s" totalImagesPullingTime="4.437740987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:35:37.056483629 +0000 UTC m=+355.879079792" watchObservedRunningTime="2026-11-03 23:35:37.056923514 +0000 UTC m=+355.879519672" Nov 03 23:35:43.044858 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:43.044827 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:43.489167 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:43.489129 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:43.629432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:43.629398 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:43.796167 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:43.796096 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lqv9x"] Nov 03 23:35:43.799513 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:43.799494 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lqv9x" Nov 03 23:35:43.810536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:43.810515 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lqv9x"] Nov 03 23:35:43.904687 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:43.904657 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lqv9x"] Nov 03 23:35:43.904868 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:35:43.904846 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mgx8p], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-lqv9x" podUID="6597facc-8d75-4b47-a85b-14189171dfc9" Nov 03 23:35:43.912839 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:43.912813 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgx8p\" (UniqueName: \"kubernetes.io/projected/6597facc-8d75-4b47-a85b-14189171dfc9-kube-api-access-mgx8p\") pod \"authorino-6f6944bcdd-lqv9x\" (UID: \"6597facc-8d75-4b47-a85b-14189171dfc9\") " pod="kuadrant-system/authorino-6f6944bcdd-lqv9x" Nov 03 23:35:44.014252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.014212 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgx8p\" (UniqueName: \"kubernetes.io/projected/6597facc-8d75-4b47-a85b-14189171dfc9-kube-api-access-mgx8p\") pod \"authorino-6f6944bcdd-lqv9x\" (UID: \"6597facc-8d75-4b47-a85b-14189171dfc9\") " pod="kuadrant-system/authorino-6f6944bcdd-lqv9x" Nov 03 23:35:44.033323 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.033294 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgx8p\" (UniqueName: \"kubernetes.io/projected/6597facc-8d75-4b47-a85b-14189171dfc9-kube-api-access-mgx8p\") pod \"authorino-6f6944bcdd-lqv9x\" (UID: \"6597facc-8d75-4b47-a85b-14189171dfc9\") " pod="kuadrant-system/authorino-6f6944bcdd-lqv9x" Nov 03 23:35:44.066388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.066306 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lqv9x" Nov 03 23:35:44.073238 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.073213 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lqv9x" Nov 03 23:35:44.114592 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.114562 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6597facc-8d75-4b47-a85b-14189171dfc9-kube-api-access-mgx8p\" (UniqueName: \"kubernetes.io/projected/6597facc-8d75-4b47-a85b-14189171dfc9-kube-api-access-mgx8p\") pod \"6597facc-8d75-4b47-a85b-14189171dfc9\" (UID: \"6597facc-8d75-4b47-a85b-14189171dfc9\") " Nov 03 23:35:44.116572 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.116540 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6597facc-8d75-4b47-a85b-14189171dfc9-kube-api-access-mgx8p" pod "6597facc-8d75-4b47-a85b-14189171dfc9" (UID: "6597facc-8d75-4b47-a85b-14189171dfc9"). InnerVolumeSpecName "kube-api-access-mgx8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:35:44.215741 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.215704 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgx8p\" (UniqueName: \"kubernetes.io/projected/6597facc-8d75-4b47-a85b-14189171dfc9-kube-api-access-mgx8p\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:35:44.235852 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.235823 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65f4bfdd95-gkr2t"] Nov 03 23:35:44.238996 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.238980 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:35:44.243567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.243545 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 23:35:44.248640 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.248616 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f4bfdd95-gkr2t"] Nov 03 23:35:44.316599 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.316495 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnshf\" (UniqueName: \"kubernetes.io/projected/3724443e-fc95-4b1d-9e43-15961aad6bdb-kube-api-access-bnshf\") pod \"authorino-65f4bfdd95-gkr2t\" (UID: \"3724443e-fc95-4b1d-9e43-15961aad6bdb\") " pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:35:44.316756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.316656 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3724443e-fc95-4b1d-9e43-15961aad6bdb-tls-cert\") pod \"authorino-65f4bfdd95-gkr2t\" (UID: \"3724443e-fc95-4b1d-9e43-15961aad6bdb\") " pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:35:44.417468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.417417 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3724443e-fc95-4b1d-9e43-15961aad6bdb-tls-cert\") pod \"authorino-65f4bfdd95-gkr2t\" (UID: \"3724443e-fc95-4b1d-9e43-15961aad6bdb\") " pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:35:44.417662 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.417487 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnshf\" (UniqueName: \"kubernetes.io/projected/3724443e-fc95-4b1d-9e43-15961aad6bdb-kube-api-access-bnshf\") pod \"authorino-65f4bfdd95-gkr2t\" (UID: \"3724443e-fc95-4b1d-9e43-15961aad6bdb\") " pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:35:44.419850 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.419830 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3724443e-fc95-4b1d-9e43-15961aad6bdb-tls-cert\") pod \"authorino-65f4bfdd95-gkr2t\" (UID: \"3724443e-fc95-4b1d-9e43-15961aad6bdb\") " pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:35:44.430536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.430515 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnshf\" (UniqueName: \"kubernetes.io/projected/3724443e-fc95-4b1d-9e43-15961aad6bdb-kube-api-access-bnshf\") pod \"authorino-65f4bfdd95-gkr2t\" (UID: \"3724443e-fc95-4b1d-9e43-15961aad6bdb\") " pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:35:44.550451 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.550418 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:35:44.824430 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.824290 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f4bfdd95-gkr2t"] Nov 03 23:35:44.827020 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:35:44.826991 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3724443e_fc95_4b1d_9e43_15961aad6bdb.slice/crio-0e072ca4ae09f5788c6bc968da77e241a33203d523221023d5221e20f507520d WatchSource:0}: Error finding container 0e072ca4ae09f5788c6bc968da77e241a33203d523221023d5221e20f507520d: Status 404 returned error can't find the container with id 0e072ca4ae09f5788c6bc968da77e241a33203d523221023d5221e20f507520d Nov 03 23:35:44.966310 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:44.966277 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:45.072539 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:45.072510 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lqv9x" Nov 03 23:35:45.073027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:45.072542 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" event={"ID":"3724443e-fc95-4b1d-9e43-15961aad6bdb","Type":"ContainerStarted","Data":"0e072ca4ae09f5788c6bc968da77e241a33203d523221023d5221e20f507520d"} Nov 03 23:35:45.107387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:45.107350 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lqv9x"] Nov 03 23:35:45.109888 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:45.109861 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lqv9x"] Nov 03 23:35:45.292086 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:45.292003 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6597facc-8d75-4b47-a85b-14189171dfc9" path="/var/lib/kubelet/pods/6597facc-8d75-4b47-a85b-14189171dfc9/volumes" Nov 03 23:35:46.080405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:46.080370 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" event={"ID":"3724443e-fc95-4b1d-9e43-15961aad6bdb","Type":"ContainerStarted","Data":"9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410"} Nov 03 23:35:46.104051 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:46.103968 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" podStartSLOduration=1.304755792 podStartE2EDuration="2.103947624s" totalImagesPullingTime="799.191832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:35:46.101714511 +0000 UTC m=+364.924310676" watchObservedRunningTime="2026-11-03 23:35:46.103947624 +0000 UTC m=+364.926543789" Nov 03 23:35:46.133684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:46.133404 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6r6m6"] Nov 03 23:35:46.133684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:46.133628 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" podUID="e27123f6-c921-4f59-afaf-57e64e15bfd6" containerName="authorino" containerID="cri-o://f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c" gracePeriod=30 Nov 03 23:35:46.374281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:46.374255 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" Nov 03 23:35:46.438589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:46.438563 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27123f6-c921-4f59-afaf-57e64e15bfd6-kube-api-access-6h9b7\" (UniqueName: \"kubernetes.io/projected/e27123f6-c921-4f59-afaf-57e64e15bfd6-kube-api-access-6h9b7\") pod \"e27123f6-c921-4f59-afaf-57e64e15bfd6\" (UID: \"e27123f6-c921-4f59-afaf-57e64e15bfd6\") " Nov 03 23:35:46.440648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:46.440625 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27123f6-c921-4f59-afaf-57e64e15bfd6-kube-api-access-6h9b7" pod "e27123f6-c921-4f59-afaf-57e64e15bfd6" (UID: "e27123f6-c921-4f59-afaf-57e64e15bfd6"). InnerVolumeSpecName "kube-api-access-6h9b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:35:46.539497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:46.539464 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h9b7\" (UniqueName: \"kubernetes.io/projected/e27123f6-c921-4f59-afaf-57e64e15bfd6-kube-api-access-6h9b7\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:35:47.086232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.086198 6208 generic.go:356] "Generic (PLEG): container finished" podID="e27123f6-c921-4f59-afaf-57e64e15bfd6" containerID="f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c" exitCode=0 Nov 03 23:35:47.086700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.086247 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" Nov 03 23:35:47.086700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.086290 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" event={"ID":"e27123f6-c921-4f59-afaf-57e64e15bfd6","Type":"ContainerDied","Data":"f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c"} Nov 03 23:35:47.086700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.086339 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6r6m6" event={"ID":"e27123f6-c921-4f59-afaf-57e64e15bfd6","Type":"ContainerDied","Data":"da5e38e332d63e4dc62ca8fcd5676aebb61e7d80dcc7482104e8feaad507e57c"} Nov 03 23:35:47.086700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.086361 6208 scope.go:122] "RemoveContainer" containerID="f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c" Nov 03 23:35:47.095674 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.095652 6208 scope.go:122] "RemoveContainer" containerID="f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c" Nov 03 23:35:47.095939 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:35:47.095918 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c\": container with ID starting with f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c not found: ID does not exist" containerID="f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c" Nov 03 23:35:47.095984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.095950 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c"} err="failed to get container status \"f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c\": rpc error: code = NotFound desc = could not find container \"f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c\": container with ID starting with f409921af8f3a089d785dbeb6cac6cccc469d3fc550e0854458e3e21da08f84c not found: ID does not exist" Nov 03 23:35:47.117989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.117955 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6r6m6"] Nov 03 23:35:47.125783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.125752 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6r6m6"] Nov 03 23:35:47.292184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:47.292152 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27123f6-c921-4f59-afaf-57e64e15bfd6" path="/var/lib/kubelet/pods/e27123f6-c921-4f59-afaf-57e64e15bfd6/volumes" Nov 03 23:35:49.932419 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:49.932335 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:35:53.742123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:53.742092 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7"] Nov 03 23:35:53.742577 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:53.742322 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" podUID="e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" containerName="maas-api" containerID="cri-o://1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424" gracePeriod=30 Nov 03 23:35:53.982989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:53.982965 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:54.105000 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.104890 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-metrics-certs\") pod \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " Nov 03 23:35:54.105000 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.104972 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-kube-api-access-92wmm\" (UniqueName: \"kubernetes.io/projected/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-kube-api-access-92wmm\") pod \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " Nov 03 23:35:54.105257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.105017 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-maas-api-tls\") pod \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " Nov 03 23:35:54.105257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.105087 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-openshift-service-ca\") pod \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\" (UID: \"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8\") " Nov 03 23:35:54.105500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.105472 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-openshift-service-ca" pod "e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" (UID: "e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:35:54.107348 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.107317 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-kube-api-access-92wmm" pod "e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" (UID: "e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8"). InnerVolumeSpecName "kube-api-access-92wmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:35:54.107437 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.107376 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-metrics-certs" pod "e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" (UID: "e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:35:54.107502 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.107430 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-maas-api-tls" pod "e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" (UID: "e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:35:54.120886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.120857 6208 generic.go:356] "Generic (PLEG): container finished" podID="e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" containerID="1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424" exitCode=0 Nov 03 23:35:54.120965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.120907 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" Nov 03 23:35:54.120965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.120935 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" event={"ID":"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8","Type":"ContainerDied","Data":"1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424"} Nov 03 23:35:54.121056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.120992 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7" event={"ID":"e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8","Type":"ContainerDied","Data":"2c4569487ec229b4ed398d17b8f19b4fa7899f287b304b335c3b81d953c0e741"} Nov 03 23:35:54.121056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.121015 6208 scope.go:122] "RemoveContainer" containerID="1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424" Nov 03 23:35:54.129569 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.129551 6208 scope.go:122] "RemoveContainer" containerID="1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424" Nov 03 23:35:54.129808 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:35:54.129788 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424\": container with ID starting with 1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424 not found: ID does not exist" containerID="1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424" Nov 03 23:35:54.129867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.129819 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424"} err="failed to get container status \"1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424\": rpc error: code = NotFound desc = could not find container \"1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424\": container with ID starting with 1c8cbd1d4b2abcfbc3fcf6d6b80cd9b7bae1cecb7bc9a5c7af0b8f346f906424 not found: ID does not exist" Nov 03 23:35:54.142968 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.142936 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7"] Nov 03 23:35:54.147280 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.147259 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-56cc74d699-9b9p7"] Nov 03 23:35:54.205922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.205892 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:35:54.205922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.205920 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:35:54.206114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.205954 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:35:54.206114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:54.205965 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92wmm\" (UniqueName: \"kubernetes.io/projected/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8-kube-api-access-92wmm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:35:55.292267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:35:55.292237 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" path="/var/lib/kubelet/pods/e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8/volumes" Nov 03 23:36:05.287390 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:05.287346 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:36:05.385515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:05.385475 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:36:05.702613 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:36:05.702532 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:36:35.704108 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:36:35.704071 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:36:45.087960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:45.087923 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:36:50.055983 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:50.055949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:36:51.484592 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.484552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:36:51.509061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.509017 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-gjj4r"] Nov 03 23:36:51.509372 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.509359 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e27123f6-c921-4f59-afaf-57e64e15bfd6" containerName="authorino" Nov 03 23:36:51.509415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.509373 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27123f6-c921-4f59-afaf-57e64e15bfd6" containerName="authorino" Nov 03 23:36:51.509415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.509411 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" containerName="maas-api" Nov 03 23:36:51.509476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.509416 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" containerName="maas-api" Nov 03 23:36:51.509476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.509473 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27123f6-c921-4f59-afaf-57e64e15bfd6" containerName="authorino" Nov 03 23:36:51.509540 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.509489 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ad1919-14dd-4a50-a2cd-b55d8e9dddf8" containerName="maas-api" Nov 03 23:36:51.512483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.512459 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:36:51.519047 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.519014 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 23:36:51.519223 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.519203 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 23:36:51.520825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.520510 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 23:36:51.520825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.520692 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 23:36:51.520825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.520776 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9qv48\"" Nov 03 23:36:51.532833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.532809 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-gjj4r"] Nov 03 23:36:51.576585 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.576557 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-gjj4r\" (UID: \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:36:51.576730 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.576603 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmkz\" (UniqueName: \"kubernetes.io/projected/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-kube-api-access-spmkz\") pod \"ai-gateway-controller-5d8dddb676-gjj4r\" (UID: \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:36:51.629713 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.629680 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:36:51.677664 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.677633 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-gjj4r\" (UID: \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:36:51.677786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.677695 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmkz\" (UniqueName: \"kubernetes.io/projected/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-kube-api-access-spmkz\") pod \"ai-gateway-controller-5d8dddb676-gjj4r\" (UID: \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:36:51.680110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.680084 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-gjj4r\" (UID: \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:36:51.690397 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.690375 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmkz\" (UniqueName: \"kubernetes.io/projected/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-kube-api-access-spmkz\") pod \"ai-gateway-controller-5d8dddb676-gjj4r\" (UID: \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:36:51.827485 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:51.827385 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:36:52.102673 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:52.102647 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-gjj4r"] Nov 03 23:36:52.104598 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:36:52.104566 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cf0e8f_e9e8_4f06_9e6d_095a59d32fc6.slice/crio-d78085474ba3c7260502be9143d23180f73bfa9ae3920ad894b7714faba2561f WatchSource:0}: Error finding container d78085474ba3c7260502be9143d23180f73bfa9ae3920ad894b7714faba2561f: Status 404 returned error can't find the container with id d78085474ba3c7260502be9143d23180f73bfa9ae3920ad894b7714faba2561f Nov 03 23:36:52.376309 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:52.376225 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" event={"ID":"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6","Type":"ContainerStarted","Data":"d78085474ba3c7260502be9143d23180f73bfa9ae3920ad894b7714faba2561f"} Nov 03 23:36:54.387208 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:54.387171 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" event={"ID":"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6","Type":"ContainerStarted","Data":"bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b"} Nov 03 23:36:54.387620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:54.387304 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:36:54.424478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:54.424408 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" podStartSLOduration=1.643443169 podStartE2EDuration="3.424390215s" totalImagesPullingTime="1.780947046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:36:54.423306086 +0000 UTC m=+433.245902247" watchObservedRunningTime="2026-11-03 23:36:54.424390215 +0000 UTC m=+433.246986385" Nov 03 23:36:55.683077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.683021 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:36:55.723944 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.723907 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx"] Nov 03 23:36:55.727402 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.727377 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:36:55.728681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.728659 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-4dsdb"] Nov 03 23:36:55.733899 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.733415 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:36:55.737073 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.736984 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 23:36:55.737210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.737161 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qkns9\"" Nov 03 23:36:55.740029 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.739971 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx"] Nov 03 23:36:55.743707 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.743681 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-4dsdb"] Nov 03 23:36:55.791172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.791144 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:36:55.812937 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.812899 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5a1e6-cb54-4115-ae52-3278607679c5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w2ckx\" (UID: \"a0e5a1e6-cb54-4115-ae52-3278607679c5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:36:55.812937 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.812941 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07df26b-e714-4555-a633-fa616bb9be2e-plugins-config-volume\") pod \"payload-processing-7bb9489676-4dsdb\" (UID: \"c07df26b-e714-4555-a633-fa616bb9be2e\") " pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:36:55.813198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.813091 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bfs\" (UniqueName: \"kubernetes.io/projected/a0e5a1e6-cb54-4115-ae52-3278607679c5-kube-api-access-t9bfs\") pod \"payload-pre-processing-7b98bb678c-w2ckx\" (UID: \"a0e5a1e6-cb54-4115-ae52-3278607679c5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:36:55.813198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.813143 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65hx\" (UniqueName: \"kubernetes.io/projected/c07df26b-e714-4555-a633-fa616bb9be2e-kube-api-access-k65hx\") pod \"payload-processing-7bb9489676-4dsdb\" (UID: \"c07df26b-e714-4555-a633-fa616bb9be2e\") " pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:36:55.913778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.913741 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bfs\" (UniqueName: \"kubernetes.io/projected/a0e5a1e6-cb54-4115-ae52-3278607679c5-kube-api-access-t9bfs\") pod \"payload-pre-processing-7b98bb678c-w2ckx\" (UID: \"a0e5a1e6-cb54-4115-ae52-3278607679c5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:36:55.913966 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.913794 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65hx\" (UniqueName: \"kubernetes.io/projected/c07df26b-e714-4555-a633-fa616bb9be2e-kube-api-access-k65hx\") pod \"payload-processing-7bb9489676-4dsdb\" (UID: \"c07df26b-e714-4555-a633-fa616bb9be2e\") " pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:36:55.913966 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.913848 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5a1e6-cb54-4115-ae52-3278607679c5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w2ckx\" (UID: \"a0e5a1e6-cb54-4115-ae52-3278607679c5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:36:55.913966 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.913872 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07df26b-e714-4555-a633-fa616bb9be2e-plugins-config-volume\") pod \"payload-processing-7bb9489676-4dsdb\" (UID: \"c07df26b-e714-4555-a633-fa616bb9be2e\") " pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:36:55.914746 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.914718 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07df26b-e714-4555-a633-fa616bb9be2e-plugins-config-volume\") pod \"payload-processing-7bb9489676-4dsdb\" (UID: \"c07df26b-e714-4555-a633-fa616bb9be2e\") " pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:36:55.914838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.914721 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5a1e6-cb54-4115-ae52-3278607679c5-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w2ckx\" (UID: \"a0e5a1e6-cb54-4115-ae52-3278607679c5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:36:55.926989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.926958 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65hx\" (UniqueName: \"kubernetes.io/projected/c07df26b-e714-4555-a633-fa616bb9be2e-kube-api-access-k65hx\") pod \"payload-processing-7bb9489676-4dsdb\" (UID: \"c07df26b-e714-4555-a633-fa616bb9be2e\") " pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:36:55.927141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:55.927067 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bfs\" (UniqueName: \"kubernetes.io/projected/a0e5a1e6-cb54-4115-ae52-3278607679c5-kube-api-access-t9bfs\") pod \"payload-pre-processing-7b98bb678c-w2ckx\" (UID: \"a0e5a1e6-cb54-4115-ae52-3278607679c5\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:36:56.049456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:56.049416 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:36:56.056735 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:56.056711 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:36:56.327647 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:56.327574 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx"] Nov 03 23:36:56.331002 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:36:56.330968 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e5a1e6_cb54_4115_ae52_3278607679c5.slice/crio-daa63e1d1f12ca378c64200a9621600c820c01c786cc55e2a2ded5d307c71d85 WatchSource:0}: Error finding container daa63e1d1f12ca378c64200a9621600c820c01c786cc55e2a2ded5d307c71d85: Status 404 returned error can't find the container with id daa63e1d1f12ca378c64200a9621600c820c01c786cc55e2a2ded5d307c71d85 Nov 03 23:36:56.341866 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:56.341841 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-4dsdb"] Nov 03 23:36:56.342595 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:36:56.342573 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07df26b_e714_4555_a633_fa616bb9be2e.slice/crio-fc97e6d5c45324b46dc8f35b6261164f07fd82dfad1157aaec3e0f91c7eaa2df WatchSource:0}: Error finding container fc97e6d5c45324b46dc8f35b6261164f07fd82dfad1157aaec3e0f91c7eaa2df: Status 404 returned error can't find the container with id fc97e6d5c45324b46dc8f35b6261164f07fd82dfad1157aaec3e0f91c7eaa2df Nov 03 23:36:56.398388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:56.398355 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" event={"ID":"c07df26b-e714-4555-a633-fa616bb9be2e","Type":"ContainerStarted","Data":"fc97e6d5c45324b46dc8f35b6261164f07fd82dfad1157aaec3e0f91c7eaa2df"} Nov 03 23:36:56.399530 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:56.399507 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" event={"ID":"a0e5a1e6-cb54-4115-ae52-3278607679c5","Type":"ContainerStarted","Data":"daa63e1d1f12ca378c64200a9621600c820c01c786cc55e2a2ded5d307c71d85"} Nov 03 23:36:59.417611 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:59.417570 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" event={"ID":"c07df26b-e714-4555-a633-fa616bb9be2e","Type":"ContainerStarted","Data":"2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db"} Nov 03 23:36:59.418108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:59.417638 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:36:59.419171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:59.419142 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" event={"ID":"a0e5a1e6-cb54-4115-ae52-3278607679c5","Type":"ContainerStarted","Data":"2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f"} Nov 03 23:36:59.419276 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:59.419210 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:36:59.438209 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:59.438166 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" podStartSLOduration=2.110422741 podStartE2EDuration="4.438151714s" totalImagesPullingTime="2.327728973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:36:59.436793596 +0000 UTC m=+438.259389758" watchObservedRunningTime="2026-11-03 23:36:59.438151714 +0000 UTC m=+438.260747869" Nov 03 23:36:59.456410 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:36:59.456365 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" podStartSLOduration=2.15498129 podStartE2EDuration="4.456352423s" totalImagesPullingTime="2.301371133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:36:59.455065877 +0000 UTC m=+438.277662040" watchObservedRunningTime="2026-11-03 23:36:59.456352423 +0000 UTC m=+438.278948582" Nov 03 23:37:05.394600 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.394571 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:37:05.681865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.681789 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6db599696d-4njr8"] Nov 03 23:37:05.685350 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.685327 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:05.695073 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.695017 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6db599696d-4njr8"] Nov 03 23:37:05.708095 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:37:05.708065 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:37:05.803657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.803620 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwp8p\" (UniqueName: \"kubernetes.io/projected/c9bf53bb-bfda-4935-8536-dd79e6e7c6c7-kube-api-access-wwp8p\") pod \"ai-gateway-controller-6db599696d-4njr8\" (UID: \"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7\") " pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:05.803825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.803691 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bf53bb-bfda-4935-8536-dd79e6e7c6c7-metrics-certs\") pod \"ai-gateway-controller-6db599696d-4njr8\" (UID: \"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7\") " pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:05.904823 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.904793 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bf53bb-bfda-4935-8536-dd79e6e7c6c7-metrics-certs\") pod \"ai-gateway-controller-6db599696d-4njr8\" (UID: \"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7\") " pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:05.904966 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.904850 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwp8p\" (UniqueName: \"kubernetes.io/projected/c9bf53bb-bfda-4935-8536-dd79e6e7c6c7-kube-api-access-wwp8p\") pod \"ai-gateway-controller-6db599696d-4njr8\" (UID: \"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7\") " pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:05.907322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.907301 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9bf53bb-bfda-4935-8536-dd79e6e7c6c7-metrics-certs\") pod \"ai-gateway-controller-6db599696d-4njr8\" (UID: \"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7\") " pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:05.918804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.918778 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwp8p\" (UniqueName: \"kubernetes.io/projected/c9bf53bb-bfda-4935-8536-dd79e6e7c6c7-kube-api-access-wwp8p\") pod \"ai-gateway-controller-6db599696d-4njr8\" (UID: \"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7\") " pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:05.999833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:05.999800 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:06.275838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:06.275652 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6db599696d-4njr8"] Nov 03 23:37:06.278343 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:37:06.278307 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bf53bb_bfda_4935_8536_dd79e6e7c6c7.slice/crio-041169aed04a485750c55bf389336903be1b5f5d522dc02559c1d3fb054918de WatchSource:0}: Error finding container 041169aed04a485750c55bf389336903be1b5f5d522dc02559c1d3fb054918de: Status 404 returned error can't find the container with id 041169aed04a485750c55bf389336903be1b5f5d522dc02559c1d3fb054918de Nov 03 23:37:06.453468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:06.453433 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" event={"ID":"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7","Type":"ContainerStarted","Data":"041169aed04a485750c55bf389336903be1b5f5d522dc02559c1d3fb054918de"} Nov 03 23:37:07.459864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:07.459828 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" event={"ID":"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7","Type":"ContainerStarted","Data":"5ab4469b3f7e503fd3bb2dc653ed559796e9cdf7d1f6992e94f58e43b521aa3f"} Nov 03 23:37:07.460303 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:07.459983 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:07.480095 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:07.480029 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" podStartSLOduration=2.09291243 podStartE2EDuration="2.48001696s" totalImagesPullingTime="387.10453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:07.479340605 +0000 UTC m=+446.301936766" watchObservedRunningTime="2026-11-03 23:37:07.48001696 +0000 UTC m=+446.302613122" Nov 03 23:37:10.425887 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:10.425859 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 03 23:37:10.426297 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:10.425924 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 03 23:37:18.466640 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:18.466605 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 03 23:37:18.523260 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:18.523226 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-gjj4r"] Nov 03 23:37:18.523460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:18.523415 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" podUID="e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6" containerName="manager" containerID="cri-o://bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b" gracePeriod=30 Nov 03 23:37:18.769731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:18.769706 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:37:18.916731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:18.916690 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-kube-api-access-spmkz\" (UniqueName: \"kubernetes.io/projected/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-kube-api-access-spmkz\") pod \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\" (UID: \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\") " Nov 03 23:37:18.916933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:18.916785 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-metrics-certs\") pod \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\" (UID: \"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6\") " Nov 03 23:37:18.918943 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:18.918915 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-metrics-certs" pod "e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6" (UID: "e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:37:18.919048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:18.919000 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-kube-api-access-spmkz" pod "e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6" (UID: "e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6"). InnerVolumeSpecName "kube-api-access-spmkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:37:19.017650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.017571 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spmkz\" (UniqueName: \"kubernetes.io/projected/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-kube-api-access-spmkz\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:37:19.017650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.017604 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:37:19.517531 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.517491 6208 generic.go:356] "Generic (PLEG): container finished" podID="e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6" containerID="bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b" exitCode=0 Nov 03 23:37:19.518005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.517555 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" Nov 03 23:37:19.518005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.517575 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" event={"ID":"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6","Type":"ContainerDied","Data":"bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b"} Nov 03 23:37:19.518005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.517669 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-gjj4r" event={"ID":"e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6","Type":"ContainerDied","Data":"d78085474ba3c7260502be9143d23180f73bfa9ae3920ad894b7714faba2561f"} Nov 03 23:37:19.518005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.517705 6208 scope.go:122] "RemoveContainer" containerID="bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b" Nov 03 23:37:19.526282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.526113 6208 scope.go:122] "RemoveContainer" containerID="bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b" Nov 03 23:37:19.526433 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:37:19.526410 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b\": container with ID starting with bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b not found: ID does not exist" containerID="bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b" Nov 03 23:37:19.526477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.526441 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b"} err="failed to get container status \"bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b\": rpc error: code = NotFound desc = could not find container \"bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b\": container with ID starting with bdff3801e1ec4ede81d8c4fe3a9d17b5a1a9fcf83bc68352752cead7751d330b not found: ID does not exist" Nov 03 23:37:19.538594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.538564 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-gjj4r"] Nov 03 23:37:19.545736 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:19.545714 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-gjj4r"] Nov 03 23:37:21.292828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:21.292795 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6" path="/var/lib/kubelet/pods/e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6/volumes" Nov 03 23:37:30.239990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:30.239950 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:30.722530 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:30.722495 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:31.929367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:31.929330 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:32.060585 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:32.060545 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:35.450321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:35.450286 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:35.705850 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:37:35.705805 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:37:38.034879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.034843 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc"] Nov 03 23:37:38.035365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.035344 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6" containerName="manager" Nov 03 23:37:38.035365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.035360 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6" containerName="manager" Nov 03 23:37:38.035490 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.035478 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cf0e8f-e9e8-4f06-9e6d-095a59d32fc6" containerName="manager" Nov 03 23:37:38.042436 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.042414 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.046722 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.046690 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 23:37:38.047735 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.047708 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vfcjl\"" Nov 03 23:37:38.047897 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.047799 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 23:37:38.047897 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.047872 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 23:37:38.048056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.048021 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc"] Nov 03 23:37:38.068024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.067992 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.068024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.068024 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.068228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.068084 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.068228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.068182 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwpk\" (UniqueName: \"kubernetes.io/projected/5c51275a-7e33-4a5c-9300-7e5e8d92c294-kube-api-access-lwwpk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.068313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.068230 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.068313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.068253 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c51275a-7e33-4a5c-9300-7e5e8d92c294-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.094816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.094783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:38.168797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.168761 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwpk\" (UniqueName: \"kubernetes.io/projected/5c51275a-7e33-4a5c-9300-7e5e8d92c294-kube-api-access-lwwpk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.168957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.168818 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.168957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.168844 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c51275a-7e33-4a5c-9300-7e5e8d92c294-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.168957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.168870 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.168957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.168890 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.168957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.168925 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.169395 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.169371 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.169395 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.169388 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.169508 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.169447 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.171363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.171340 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c51275a-7e33-4a5c-9300-7e5e8d92c294-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.171532 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.171514 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c51275a-7e33-4a5c-9300-7e5e8d92c294-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.180783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.180765 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwpk\" (UniqueName: \"kubernetes.io/projected/5c51275a-7e33-4a5c-9300-7e5e8d92c294-kube-api-access-lwwpk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qq6xc\" (UID: \"5c51275a-7e33-4a5c-9300-7e5e8d92c294\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.358894 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.358814 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:38.636997 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.636927 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc"] Nov 03 23:37:38.640399 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:37:38.640372 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c51275a_7e33_4a5c_9300_7e5e8d92c294.slice/crio-0933891dd6968c31e812421da48a7dd0e710bc7ffb33c507d0f8af01319f74d6 WatchSource:0}: Error finding container 0933891dd6968c31e812421da48a7dd0e710bc7ffb33c507d0f8af01319f74d6: Status 404 returned error can't find the container with id 0933891dd6968c31e812421da48a7dd0e710bc7ffb33c507d0f8af01319f74d6 Nov 03 23:37:38.642109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.642089 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:37:38.763998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:38.763955 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:39.434821 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.434787 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th"] Nov 03 23:37:39.439718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.439688 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.449078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.449017 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 23:37:39.449367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.449341 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th"] Nov 03 23:37:39.479060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.479004 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:39.482238 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.482209 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.482375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.482263 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twjvs\" (UniqueName: \"kubernetes.io/projected/ef180863-1732-423f-a932-989ba01cb99c-kube-api-access-twjvs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.482375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.482291 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.482375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.482349 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.482499 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.482432 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.482499 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.482447 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef180863-1732-423f-a932-989ba01cb99c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.582960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.582926 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.582960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.582973 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef180863-1732-423f-a932-989ba01cb99c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.583240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.583002 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.583240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.583055 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twjvs\" (UniqueName: \"kubernetes.io/projected/ef180863-1732-423f-a932-989ba01cb99c-kube-api-access-twjvs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.583240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.583083 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.583240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.583118 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.583454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.583438 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.583687 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.583667 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.584182 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.584071 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.585655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.585601 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef180863-1732-423f-a932-989ba01cb99c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.589149 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.589125 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef180863-1732-423f-a932-989ba01cb99c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.596636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.596616 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twjvs\" (UniqueName: \"kubernetes.io/projected/ef180863-1732-423f-a932-989ba01cb99c-kube-api-access-twjvs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fj7th\" (UID: \"ef180863-1732-423f-a932-989ba01cb99c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:39.615026 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.614990 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" event={"ID":"5c51275a-7e33-4a5c-9300-7e5e8d92c294","Type":"ContainerStarted","Data":"0933891dd6968c31e812421da48a7dd0e710bc7ffb33c507d0f8af01319f74d6"} Nov 03 23:37:39.762279 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:39.762241 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:40.068056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:40.064058 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th"] Nov 03 23:37:40.407903 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:40.407863 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:40.621504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:40.621463 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" event={"ID":"5c51275a-7e33-4a5c-9300-7e5e8d92c294","Type":"ContainerStarted","Data":"a2e6cf43c911b33b41dc40e9a5fdf774be674f3488b683aca28e1bdb7e7ff98a"} Nov 03 23:37:40.623083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:40.623056 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" event={"ID":"ef180863-1732-423f-a932-989ba01cb99c","Type":"ContainerStarted","Data":"9078ddf35758944c3c282455b18e0ab29aa44b5b392f2ea5863db87dfa62e776"} Nov 03 23:37:40.623177 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:40.623107 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" event={"ID":"ef180863-1732-423f-a932-989ba01cb99c","Type":"ContainerStarted","Data":"1db985bc2169abd2a771f73174953a51ed734f5852901d5ca7aa95ca0ce3ef46"} Nov 03 23:37:40.643657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:40.643572 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" podStartSLOduration=1.294417827 podStartE2EDuration="2.643557791s" totalImagesPullingTime="1.349139964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:40.642411338 +0000 UTC m=+479.465007500" watchObservedRunningTime="2026-11-03 23:37:40.643557791 +0000 UTC m=+479.466153952" Nov 03 23:37:40.663847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:40.663800 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" podStartSLOduration=1.37519187 podStartE2EDuration="1.663788067s" totalImagesPullingTime="288.596197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:40.662633237 +0000 UTC m=+479.485229400" watchObservedRunningTime="2026-11-03 23:37:40.663788067 +0000 UTC m=+479.486384229" Nov 03 23:37:40.763201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:40.763168 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:41.359921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:41.359893 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:42.296645 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.296613 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lg6nd"] Nov 03 23:37:42.301388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.301364 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.309559 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.309532 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-home\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.309689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.309581 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c22dc8-99b1-4cbd-b15a-c7d48742940a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.309689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.309660 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.309781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.309692 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.309781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.309716 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.309781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.309737 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kxs\" (UniqueName: \"kubernetes.io/projected/83c22dc8-99b1-4cbd-b15a-c7d48742940a-kube-api-access-v6kxs\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.312594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.312572 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 23:37:42.315866 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.315840 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lg6nd"] Nov 03 23:37:42.379102 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.379067 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:42.411102 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.411061 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.411282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.411118 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.411282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.411152 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.411282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.411177 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kxs\" (UniqueName: \"kubernetes.io/projected/83c22dc8-99b1-4cbd-b15a-c7d48742940a-kube-api-access-v6kxs\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.411282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.411242 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-home\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.411492 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.411292 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c22dc8-99b1-4cbd-b15a-c7d48742940a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.411573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.411550 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.411725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.411702 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.412020 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.411992 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-home\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.413838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.413808 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c22dc8-99b1-4cbd-b15a-c7d48742940a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.413915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.413848 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c22dc8-99b1-4cbd-b15a-c7d48742940a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.423758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.423736 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kxs\" (UniqueName: \"kubernetes.io/projected/83c22dc8-99b1-4cbd-b15a-c7d48742940a-kube-api-access-v6kxs\") pod \"echo-simulator-kserve-64b6f8f96c-lg6nd\" (UID: \"83c22dc8-99b1-4cbd-b15a-c7d48742940a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.615260 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.615159 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:42.910280 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.904362 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lg6nd"] Nov 03 23:37:42.999938 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:42.999899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:43.647882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:43.647842 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" event={"ID":"83c22dc8-99b1-4cbd-b15a-c7d48742940a","Type":"ContainerStarted","Data":"2054d8fe4837055b5a1672a387accc66ca01db291d5946b7084c216eb25d912b"} Nov 03 23:37:44.415517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:44.415479 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:45.114366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:45.114280 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:45.659423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:45.659391 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" event={"ID":"83c22dc8-99b1-4cbd-b15a-c7d48742940a","Type":"ContainerStarted","Data":"a790c8a4d7431391348b0e1124f32e56b73967619a3199cd5f0ef38cba3ab787"} Nov 03 23:37:45.679995 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:45.679943 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" podStartSLOduration=1.765939657 podStartE2EDuration="3.6799297s" totalImagesPullingTime="1.913990043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:45.679429011 +0000 UTC m=+484.502025173" watchObservedRunningTime="2026-11-03 23:37:45.6799297 +0000 UTC m=+484.502525861" Nov 03 23:37:46.615987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:46.615942 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:47.274641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.274599 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd"] Nov 03 23:37:47.278565 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.278535 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.283105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.283079 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd"] Nov 03 23:37:47.283658 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.283634 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 23:37:47.285589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.285549 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sh6t7\"" Nov 03 23:37:47.363661 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.363627 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tbd\" (UniqueName: \"kubernetes.io/projected/84fcd33b-388e-4e91-a74e-fe88a560c5d8-kube-api-access-h5tbd\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.363864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.363669 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.363864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.363713 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.363864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.363839 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.364061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.363944 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.364120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.364095 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.364188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.364146 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.364708 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.364671 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.364823 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.364717 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.465365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465330 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.465365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465371 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.465629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465396 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.465629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465414 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tbd\" (UniqueName: \"kubernetes.io/projected/84fcd33b-388e-4e91-a74e-fe88a560c5d8-kube-api-access-h5tbd\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.465629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465584 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.465811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465647 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.465811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465678 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.465811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465719 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.465811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465783 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.466004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465863 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.466004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.465943 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.466179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.466109 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.466242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.466219 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.466601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.466573 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.468321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.468296 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.468407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.468368 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.504697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.504669 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tbd\" (UniqueName: \"kubernetes.io/projected/84fcd33b-388e-4e91-a74e-fe88a560c5d8-kube-api-access-h5tbd\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.505337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.505314 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84fcd33b-388e-4e91-a74e-fe88a560c5d8-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ffbsd\" (UID: \"84fcd33b-388e-4e91-a74e-fe88a560c5d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.604168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.604070 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:47.888904 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:47.888880 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd"] Nov 03 23:37:47.890788 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:37:47.890748 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fcd33b_388e_4e91_a74e_fe88a560c5d8.slice/crio-87279302ce050eb84e7e6d1b4c704a0d5276e09c3c67764c9620ac1f859f5749 WatchSource:0}: Error finding container 87279302ce050eb84e7e6d1b4c704a0d5276e09c3c67764c9620ac1f859f5749: Status 404 returned error can't find the container with id 87279302ce050eb84e7e6d1b4c704a0d5276e09c3c67764c9620ac1f859f5749 Nov 03 23:37:48.359435 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:48.359395 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:48.372169 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:48.372146 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:48.688263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:48.688173 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" event={"ID":"84fcd33b-388e-4e91-a74e-fe88a560c5d8","Type":"ContainerStarted","Data":"87279302ce050eb84e7e6d1b4c704a0d5276e09c3c67764c9620ac1f859f5749"} Nov 03 23:37:48.699443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:48.699416 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qq6xc" Nov 03 23:37:49.763356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:49.763320 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:49.777845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:49.777822 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:50.060204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:50.060125 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:37:50.712447 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:50.712423 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" Nov 03 23:37:51.273776 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:51.273744 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:37:51.274065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:51.273831 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:37:51.274065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:51.273862 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:37:51.717264 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:51.717222 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" event={"ID":"84fcd33b-388e-4e91-a74e-fe88a560c5d8","Type":"ContainerStarted","Data":"5a52cfc82b494077d6403aa832743297d7a9e4a645911a122f41e94240955cb8"} Nov 03 23:37:51.742929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:51.742874 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podStartSLOduration=1.362696296 podStartE2EDuration="4.742860346s" totalImagesPullingTime="3.38016405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:37:51.741303312 +0000 UTC m=+490.563899473" watchObservedRunningTime="2026-11-03 23:37:51.742860346 +0000 UTC m=+490.565456508" Nov 03 23:37:52.604675 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:52.604631 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:52.606321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:52.606291 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 23:37:52.606452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:52.606348 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 23:37:52.615359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:52.615336 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:52.627237 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:52.627212 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:52.736820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:52.736792 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lg6nd" Nov 03 23:37:53.605352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:53.605313 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 23:37:53.605764 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:53.605376 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 23:37:54.604636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:54.604594 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 23:37:54.604828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:54.604660 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 23:37:55.605457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:55.605419 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 23:37:55.605842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:55.605476 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 23:37:56.605105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:56.605066 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 23:37:56.605289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:56.605125 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 23:37:57.605321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:57.605273 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 23:37:57.605712 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:57.605336 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 23:37:58.604907 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:58.604870 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 03 23:37:58.605098 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:58.604947 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 03 23:37:59.609178 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:59.609148 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:59.609562 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:59.609510 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:37:59.610279 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:37:59.610259 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" Nov 03 23:38:02.318396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.318362 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7"] Nov 03 23:38:02.321873 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.321855 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.350189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.350161 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7"] Nov 03 23:38:02.432773 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.432738 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22176409-e847-4d2e-bf20-487bf70a69d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.432773 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.432778 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.433027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.432873 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.433027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.432892 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22176409-e847-4d2e-bf20-487bf70a69d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.433027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.432910 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.433027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.432928 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6s9j\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-kube-api-access-c6s9j\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.433027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.433002 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.433245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.433102 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.433245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.433195 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.534426 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534389 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6s9j\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-kube-api-access-c6s9j\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.534620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534433 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.534620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534456 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.534620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534500 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.534620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534522 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22176409-e847-4d2e-bf20-487bf70a69d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.534620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534540 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.534620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534595 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.534620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534618 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22176409-e847-4d2e-bf20-487bf70a69d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.534988 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534660 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.535076 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.534981 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.535076 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.535051 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.535187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.535092 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.535233 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.535175 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.535425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.535408 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22176409-e847-4d2e-bf20-487bf70a69d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.537425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.537407 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.537666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.537647 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22176409-e847-4d2e-bf20-487bf70a69d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.547955 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.547935 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.548268 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.548242 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6s9j\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-kube-api-access-c6s9j\") pod \"maas-default-gateway-openshift-default-778b7b5c47-lfks7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.634127 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.634028 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:02.914553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.914527 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7"] Nov 03 23:38:02.916459 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:38:02.916434 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22176409_e847_4d2e_bf20_487bf70a69d7.slice/crio-164add30b46e84a9c2f33b4a3e0aa600009d356796e5f3673d6f9af166a4424b WatchSource:0}: Error finding container 164add30b46e84a9c2f33b4a3e0aa600009d356796e5f3673d6f9af166a4424b: Status 404 returned error can't find the container with id 164add30b46e84a9c2f33b4a3e0aa600009d356796e5f3673d6f9af166a4424b Nov 03 23:38:02.919500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.919470 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:38:02.919580 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.919535 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:38:02.919580 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:02.919565 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:38:03.782507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:03.782467 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" event={"ID":"22176409-e847-4d2e-bf20-487bf70a69d7","Type":"ContainerStarted","Data":"ccc9e4a5c8771bb436f86b6ed5adf4b5b32a31be7c783b2258b68e0d64b4d32b"} Nov 03 23:38:03.782921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:03.782523 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" event={"ID":"22176409-e847-4d2e-bf20-487bf70a69d7","Type":"ContainerStarted","Data":"164add30b46e84a9c2f33b4a3e0aa600009d356796e5f3673d6f9af166a4424b"} Nov 03 23:38:03.808958 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:03.808896 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" podStartSLOduration=1.808881692 podStartE2EDuration="1.808881692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:38:03.806869894 +0000 UTC m=+502.629466058" watchObservedRunningTime="2026-11-03 23:38:03.808881692 +0000 UTC m=+502.631477914" Nov 03 23:38:04.635167 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:04.635133 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:04.636531 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:04.636509 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lfks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 23:38:04.636601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:04.636555 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 23:38:05.635178 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:05.635139 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lfks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 23:38:05.635597 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:05.635204 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 23:38:05.704368 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:38:05.704326 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:38:06.635418 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:06.635382 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lfks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 23:38:06.635806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:06.635453 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 23:38:07.634867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:07.634835 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lfks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 23:38:07.635079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:07.634899 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 23:38:08.477057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.473026 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55b594799b-mkv2k"] Nov 03 23:38:08.481452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.481424 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b594799b-mkv2k" Nov 03 23:38:08.484061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.484015 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55b594799b-mkv2k"] Nov 03 23:38:08.509789 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.509759 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93e48259-862a-4633-b927-84fd5351d25a-tls-cert\") pod \"authorino-55b594799b-mkv2k\" (UID: \"93e48259-862a-4633-b927-84fd5351d25a\") " pod="kuadrant-system/authorino-55b594799b-mkv2k" Nov 03 23:38:08.509901 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.509801 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksjtm\" (UniqueName: \"kubernetes.io/projected/93e48259-862a-4633-b927-84fd5351d25a-kube-api-access-ksjtm\") pod \"authorino-55b594799b-mkv2k\" (UID: \"93e48259-862a-4633-b927-84fd5351d25a\") " pod="kuadrant-system/authorino-55b594799b-mkv2k" Nov 03 23:38:08.610999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.610963 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93e48259-862a-4633-b927-84fd5351d25a-tls-cert\") pod \"authorino-55b594799b-mkv2k\" (UID: \"93e48259-862a-4633-b927-84fd5351d25a\") " pod="kuadrant-system/authorino-55b594799b-mkv2k" Nov 03 23:38:08.611192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.611004 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksjtm\" (UniqueName: \"kubernetes.io/projected/93e48259-862a-4633-b927-84fd5351d25a-kube-api-access-ksjtm\") pod \"authorino-55b594799b-mkv2k\" (UID: \"93e48259-862a-4633-b927-84fd5351d25a\") " pod="kuadrant-system/authorino-55b594799b-mkv2k" Nov 03 23:38:08.613462 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.613433 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93e48259-862a-4633-b927-84fd5351d25a-tls-cert\") pod \"authorino-55b594799b-mkv2k\" (UID: \"93e48259-862a-4633-b927-84fd5351d25a\") " pod="kuadrant-system/authorino-55b594799b-mkv2k" Nov 03 23:38:08.623546 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.623525 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksjtm\" (UniqueName: \"kubernetes.io/projected/93e48259-862a-4633-b927-84fd5351d25a-kube-api-access-ksjtm\") pod \"authorino-55b594799b-mkv2k\" (UID: \"93e48259-862a-4633-b927-84fd5351d25a\") " pod="kuadrant-system/authorino-55b594799b-mkv2k" Nov 03 23:38:08.634515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.634487 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lfks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 23:38:08.634620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.634535 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 23:38:08.795492 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:08.795396 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b594799b-mkv2k" Nov 03 23:38:09.069983 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:09.069942 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55b594799b-mkv2k"] Nov 03 23:38:09.072257 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:38:09.072220 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93e48259_862a_4633_b927_84fd5351d25a.slice/crio-cd007ec377261356e2c315784ea92b906d0cdabf378842401bc5066d34d3fcb1 WatchSource:0}: Error finding container cd007ec377261356e2c315784ea92b906d0cdabf378842401bc5066d34d3fcb1: Status 404 returned error can't find the container with id cd007ec377261356e2c315784ea92b906d0cdabf378842401bc5066d34d3fcb1 Nov 03 23:38:09.634798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:09.634757 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-lfks7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 23:38:09.635303 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:09.634840 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 23:38:09.815072 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:09.815018 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b594799b-mkv2k" event={"ID":"93e48259-862a-4633-b927-84fd5351d25a","Type":"ContainerStarted","Data":"cd007ec377261356e2c315784ea92b906d0cdabf378842401bc5066d34d3fcb1"} Nov 03 23:38:10.835741 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:10.835701 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b594799b-mkv2k" event={"ID":"93e48259-862a-4633-b927-84fd5351d25a","Type":"ContainerStarted","Data":"7df6164accb3dcfaaec6dd39beb3d075757f439c6978f2976691ea918c6d6d4a"} Nov 03 23:38:10.855810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:10.855758 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-55b594799b-mkv2k" podStartSLOduration=1.835908257 podStartE2EDuration="2.855743746s" totalImagesPullingTime="1.019835489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:38:10.855068996 +0000 UTC m=+509.677665156" watchObservedRunningTime="2026-11-03 23:38:10.855743746 +0000 UTC m=+509.678339907" Nov 03 23:38:10.902828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:10.902252 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65f4bfdd95-gkr2t"] Nov 03 23:38:10.902828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:10.902494 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" podUID="3724443e-fc95-4b1d-9e43-15961aad6bdb" containerName="authorino" containerID="cri-o://9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410" gracePeriod=30 Nov 03 23:38:11.199064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.198998 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:38:11.239367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.239331 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3724443e-fc95-4b1d-9e43-15961aad6bdb-tls-cert\" (UniqueName: \"kubernetes.io/secret/3724443e-fc95-4b1d-9e43-15961aad6bdb-tls-cert\") pod \"3724443e-fc95-4b1d-9e43-15961aad6bdb\" (UID: \"3724443e-fc95-4b1d-9e43-15961aad6bdb\") " Nov 03 23:38:11.239571 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.239450 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3724443e-fc95-4b1d-9e43-15961aad6bdb-kube-api-access-bnshf\" (UniqueName: \"kubernetes.io/projected/3724443e-fc95-4b1d-9e43-15961aad6bdb-kube-api-access-bnshf\") pod \"3724443e-fc95-4b1d-9e43-15961aad6bdb\" (UID: \"3724443e-fc95-4b1d-9e43-15961aad6bdb\") " Nov 03 23:38:11.242069 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.241988 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3724443e-fc95-4b1d-9e43-15961aad6bdb-kube-api-access-bnshf" pod "3724443e-fc95-4b1d-9e43-15961aad6bdb" (UID: "3724443e-fc95-4b1d-9e43-15961aad6bdb"). InnerVolumeSpecName "kube-api-access-bnshf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:38:11.256061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.256012 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3724443e-fc95-4b1d-9e43-15961aad6bdb-tls-cert" pod "3724443e-fc95-4b1d-9e43-15961aad6bdb" (UID: "3724443e-fc95-4b1d-9e43-15961aad6bdb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:38:11.347644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.345613 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnshf\" (UniqueName: \"kubernetes.io/projected/3724443e-fc95-4b1d-9e43-15961aad6bdb-kube-api-access-bnshf\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:38:11.347644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.345657 6208 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3724443e-fc95-4b1d-9e43-15961aad6bdb-tls-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:38:11.463394 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.463365 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:11.463983 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.463959 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:11.464953 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.464933 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:38:11.841624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.841591 6208 generic.go:356] "Generic (PLEG): container finished" podID="3724443e-fc95-4b1d-9e43-15961aad6bdb" containerID="9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410" exitCode=0 Nov 03 23:38:11.842116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.841654 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" Nov 03 23:38:11.842116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.841685 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" event={"ID":"3724443e-fc95-4b1d-9e43-15961aad6bdb","Type":"ContainerDied","Data":"9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410"} Nov 03 23:38:11.842116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.841737 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f4bfdd95-gkr2t" event={"ID":"3724443e-fc95-4b1d-9e43-15961aad6bdb","Type":"ContainerDied","Data":"0e072ca4ae09f5788c6bc968da77e241a33203d523221023d5221e20f507520d"} Nov 03 23:38:11.842116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.841762 6208 scope.go:122] "RemoveContainer" containerID="9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410" Nov 03 23:38:11.851143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.851085 6208 scope.go:122] "RemoveContainer" containerID="9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410" Nov 03 23:38:11.851418 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:38:11.851390 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410\": container with ID starting with 9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410 not found: ID does not exist" containerID="9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410" Nov 03 23:38:11.851493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.851420 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410"} err="failed to get container status \"9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410\": rpc error: code = NotFound desc = could not find container \"9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410\": container with ID starting with 9e25ce173cdc1d7c8bdcb9aa58cba2132a683137c296cb00e4367cd2706aa410 not found: ID does not exist" Nov 03 23:38:11.868821 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.868793 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65f4bfdd95-gkr2t"] Nov 03 23:38:11.872227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.872202 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65f4bfdd95-gkr2t"] Nov 03 23:38:11.903676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:11.903631 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:12.014710 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:12.014673 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:13.292809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:13.292776 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3724443e-fc95-4b1d-9e43-15961aad6bdb" path="/var/lib/kubelet/pods/3724443e-fc95-4b1d-9e43-15961aad6bdb/volumes" Nov 03 23:38:13.447630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:13.447596 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:13.555708 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:13.555628 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:17.337815 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.337777 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc"] Nov 03 23:38:17.338235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.338175 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3724443e-fc95-4b1d-9e43-15961aad6bdb" containerName="authorino" Nov 03 23:38:17.338235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.338188 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3724443e-fc95-4b1d-9e43-15961aad6bdb" containerName="authorino" Nov 03 23:38:17.338311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.338258 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3724443e-fc95-4b1d-9e43-15961aad6bdb" containerName="authorino" Nov 03 23:38:17.346181 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.346105 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.358641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.358617 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc"] Nov 03 23:38:17.407706 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.407672 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.407706 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.407707 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586d3517-e490-46c7-a81a-3be5ff076064-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.408004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.407982 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.408207 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.408181 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.408281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.408267 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.408337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.408316 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586d3517-e490-46c7-a81a-3be5ff076064-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.408417 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.408404 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.408458 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.408426 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4c8d\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-kube-api-access-j4c8d\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.408499 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.408458 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.508985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.508947 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.508985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.508989 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c8d\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-kube-api-access-j4c8d\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509013 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509064 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509088 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586d3517-e490-46c7-a81a-3be5ff076064-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509175 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509207 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509234 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509261 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586d3517-e490-46c7-a81a-3be5ff076064-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509566 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509603 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509713 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509658 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.509925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.509901 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.510201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.510184 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586d3517-e490-46c7-a81a-3be5ff076064-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.511942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.511918 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.512279 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.512259 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586d3517-e490-46c7-a81a-3be5ff076064-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.521783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.521752 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.521783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.521778 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4c8d\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-kube-api-access-j4c8d\") pod \"maas-default-gateway-openshift-default-778b7b5c47-g9jzc\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.661167 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.661082 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:17.948095 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.948063 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc"] Nov 03 23:38:17.949889 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:38:17.949866 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586d3517_e490_46c7_a81a_3be5ff076064.slice/crio-4839382d4f778f23ad54297e26df18df2d9ca4fc32336031b6ae58309d2d901d WatchSource:0}: Error finding container 4839382d4f778f23ad54297e26df18df2d9ca4fc32336031b6ae58309d2d901d: Status 404 returned error can't find the container with id 4839382d4f778f23ad54297e26df18df2d9ca4fc32336031b6ae58309d2d901d Nov 03 23:38:17.952199 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.952166 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:38:17.952256 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.952236 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:38:17.952295 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:17.952264 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:38:18.884131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:18.884090 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" event={"ID":"586d3517-e490-46c7-a81a-3be5ff076064","Type":"ContainerStarted","Data":"a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff"} Nov 03 23:38:18.884544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:18.884152 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" event={"ID":"586d3517-e490-46c7-a81a-3be5ff076064","Type":"ContainerStarted","Data":"4839382d4f778f23ad54297e26df18df2d9ca4fc32336031b6ae58309d2d901d"} Nov 03 23:38:18.910737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:18.910679 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" podStartSLOduration=1.910663673 podStartE2EDuration="1.910663673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:38:18.908271486 +0000 UTC m=+517.730867648" watchObservedRunningTime="2026-11-03 23:38:18.910663673 +0000 UTC m=+517.733259838" Nov 03 23:38:19.667912 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:19.667878 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:19.668270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:19.668248 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-g9jzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 23:38:19.668433 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:19.668410 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 23:38:20.675059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:20.674509 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-g9jzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 23:38:20.675059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:20.674578 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 23:38:21.673055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:21.668539 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-g9jzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 23:38:21.673055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:21.668613 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 23:38:22.672650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:22.671654 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-g9jzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 23:38:22.672650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:22.671715 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 23:38:23.672059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:23.662472 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-g9jzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 23:38:23.672059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:23.662546 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 23:38:24.662334 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:24.662299 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-g9jzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 23:38:24.662745 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:24.662361 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 23:38:26.420796 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:26.420762 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:26.421387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:26.421166 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:26.422063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:26.422016 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 03 23:38:28.088754 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:28.088634 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:31.729302 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:31.729268 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:35.703880 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:38:35.703841 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:38:38.565626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:38.565583 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:42.500332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:42.500296 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:45.168760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:45.168730 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:50.155897 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:50.155797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:53.868751 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:53.868711 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:38:53.973645 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:38:53.973614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:05.707465 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:05.707423 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:39:09.565794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:09.565757 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:09.704866 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:09.704824 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:09.827735 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:09.827646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:10.129799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.129716 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv"] Nov 03 23:39:10.133171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.133153 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.138401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.138380 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-80942cbb-openshift-default-dockercfg-lrdz9\"" Nov 03 23:39:10.149263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.149237 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv"] Nov 03 23:39:10.183746 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.183709 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:10.226943 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.226902 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-credential-socket\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.227131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.227071 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd01e-6450-44ea-987d-989cf05595a4-istio-podinfo\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.227131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.227103 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-socket\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.227219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.227174 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-certs\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.227269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.227243 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-envoy\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.227406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.227384 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-istio-token\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.227448 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.227426 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bttw\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-kube-api-access-4bttw\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.227574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.227554 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd01e-6450-44ea-987d-989cf05595a4-istiod-ca-cert\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.227649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.227588 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-data\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.248628 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.248600 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt"] Nov 03 23:39:10.252226 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.252209 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.266917 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.266894 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7c2a2d28-openshift-default-dockercfg-bqgsk\"" Nov 03 23:39:10.328632 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.328599 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-credential-socket\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.328810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.328643 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istiod-ca-cert\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.328810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.328719 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-data\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.328810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.328771 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd01e-6450-44ea-987d-989cf05595a4-istiod-ca-cert\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.328810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.328789 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-token\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.328810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.328806 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-data\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.328828 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-podinfo\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.329096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.328848 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-socket\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.329096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.328989 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-credential-socket\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329024 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggdd\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-kube-api-access-zggdd\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.329096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329075 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd01e-6450-44ea-987d-989cf05595a4-istio-podinfo\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329097 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-socket\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329114 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-certs\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329186 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-envoy\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.329425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329250 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-certs\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.329425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329393 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-envoy\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329414 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-data\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329436 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-credential-socket\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329501 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-istio-token\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329529 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bttw\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-kube-api-access-4bttw\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329547 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-certs\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329586 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-socket\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.329694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.329684 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd01e-6450-44ea-987d-989cf05595a4-istiod-ca-cert\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.331493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.331470 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-envoy\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.331716 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.331699 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd01e-6450-44ea-987d-989cf05595a4-istio-podinfo\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.352909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.352882 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bttw\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-kube-api-access-4bttw\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.360073 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.360015 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-istio-token\") pod \"e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.413200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.413101 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt"] Nov 03 23:39:10.421509 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.421479 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:10.430612 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.430579 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zggdd\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-kube-api-access-zggdd\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.430781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.430665 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-envoy\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.430781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.430710 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-certs\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.430781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.430770 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-credential-socket\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.430926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.430871 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istiod-ca-cert\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.430926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.430920 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-data\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.431002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.430954 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-token\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.431002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.430986 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-podinfo\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.431119 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.431025 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-socket\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.431216 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.431198 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-credential-socket\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.431335 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.431318 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-data\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.431418 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.431395 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-socket\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.431598 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.431580 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istiod-ca-cert\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.432265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.432245 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-certs\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.433398 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.433376 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-podinfo\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.433510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.433493 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-envoy\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.446096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.446074 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:10.451834 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.451813 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggdd\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-kube-api-access-zggdd\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.469145 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.469123 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-token\") pod \"e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.565286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.565250 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:10.568261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.568224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:10.727555 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.727528 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv"] Nov 03 23:39:10.729529 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:10.729502 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908dd01e_6450_44ea_987d_989cf05595a4.slice/crio-9b042a265815ddf3a1fccbeee0541df046e6b6f9372a7a114402a9722cd09204 WatchSource:0}: Error finding container 9b042a265815ddf3a1fccbeee0541df046e6b6f9372a7a114402a9722cd09204: Status 404 returned error can't find the container with id 9b042a265815ddf3a1fccbeee0541df046e6b6f9372a7a114402a9722cd09204 Nov 03 23:39:10.731801 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.731767 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:10.731890 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.731830 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:10.731890 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.731859 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:10.846256 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.846226 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt"] Nov 03 23:39:10.847531 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:10.847495 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa30bbe_ebf1_4dba_b206_41223aa75c1d.slice/crio-fc226a36dd19a3367148b0ac9da629ab40e59bbd8a1e4308da0bf59a4f23cc99 WatchSource:0}: Error finding container fc226a36dd19a3367148b0ac9da629ab40e59bbd8a1e4308da0bf59a4f23cc99: Status 404 returned error can't find the container with id fc226a36dd19a3367148b0ac9da629ab40e59bbd8a1e4308da0bf59a4f23cc99 Nov 03 23:39:10.849925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.849885 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:10.849995 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.849966 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:10.850051 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:10.850007 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:11.154155 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.154068 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" event={"ID":"dfa30bbe-ebf1-4dba-b206-41223aa75c1d","Type":"ContainerStarted","Data":"394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8"} Nov 03 23:39:11.154155 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.154136 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" event={"ID":"dfa30bbe-ebf1-4dba-b206-41223aa75c1d","Type":"ContainerStarted","Data":"fc226a36dd19a3367148b0ac9da629ab40e59bbd8a1e4308da0bf59a4f23cc99"} Nov 03 23:39:11.155642 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.155616 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" event={"ID":"908dd01e-6450-44ea-987d-989cf05595a4","Type":"ContainerStarted","Data":"343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba"} Nov 03 23:39:11.155754 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.155662 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" event={"ID":"908dd01e-6450-44ea-987d-989cf05595a4","Type":"ContainerStarted","Data":"9b042a265815ddf3a1fccbeee0541df046e6b6f9372a7a114402a9722cd09204"} Nov 03 23:39:11.182636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.182577 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" podStartSLOduration=1.182563121 podStartE2EDuration="1.182563121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:11.180412784 +0000 UTC m=+570.003008947" watchObservedRunningTime="2026-11-03 23:39:11.182563121 +0000 UTC m=+570.005159359" Nov 03 23:39:11.207811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.207745 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" podStartSLOduration=1.207725685 podStartE2EDuration="1.207725685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:11.206484288 +0000 UTC m=+570.029080455" watchObservedRunningTime="2026-11-03 23:39:11.207725685 +0000 UTC m=+570.030321872" Nov 03 23:39:11.446497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.446420 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:11.451234 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.451210 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:11.562545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.562509 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:11.565443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.565421 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:11.570655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:11.570634 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:12.052401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:12.052362 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:12.161609 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:12.161569 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:12.161861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:12.161655 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:12.162388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:12.162367 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:12.162520 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:12.162503 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:12.256276 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:12.256222 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:12.754854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:12.754823 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:12.908456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:12.908422 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:13.020060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:13.019936 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:14.065934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.065902 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:14.090849 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.090817 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228"] Nov 03 23:39:14.094238 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.094213 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:14.096935 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.096912 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs"] Nov 03 23:39:14.100423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.100400 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:14.115657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.115636 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7c2a2d28-dockercfg-6cqfr\"" Nov 03 23:39:14.118082 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.118059 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228"] Nov 03 23:39:14.126633 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.126607 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs"] Nov 03 23:39:14.169886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.169845 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbfb154-b4c1-4952-8423-fa03f425a559-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228\" (UID: \"abbfb154-b4c1-4952-8423-fa03f425a559\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:14.170109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.169899 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5crb\" (UniqueName: \"kubernetes.io/projected/abbfb154-b4c1-4952-8423-fa03f425a559-kube-api-access-g5crb\") pod \"payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228\" (UID: \"abbfb154-b4c1-4952-8423-fa03f425a559\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:14.185705 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.185673 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7c2a2d28\"" Nov 03 23:39:14.202933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.202901 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:14.270993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.270953 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60dca8f6-9229-4ddc-bf78-227ddd3950b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs\" (UID: \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\") " pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:14.271203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.271188 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/60dca8f6-9229-4ddc-bf78-227ddd3950b4-kube-api-access-r89vf\") pod \"payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs\" (UID: \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\") " pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:14.271256 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.271229 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbfb154-b4c1-4952-8423-fa03f425a559-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228\" (UID: \"abbfb154-b4c1-4952-8423-fa03f425a559\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:14.271933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.271904 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5crb\" (UniqueName: \"kubernetes.io/projected/abbfb154-b4c1-4952-8423-fa03f425a559-kube-api-access-g5crb\") pod \"payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228\" (UID: \"abbfb154-b4c1-4952-8423-fa03f425a559\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:14.272800 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.272581 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbfb154-b4c1-4952-8423-fa03f425a559-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228\" (UID: \"abbfb154-b4c1-4952-8423-fa03f425a559\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:14.285246 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.285211 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5crb\" (UniqueName: \"kubernetes.io/projected/abbfb154-b4c1-4952-8423-fa03f425a559-kube-api-access-g5crb\") pod \"payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228\" (UID: \"abbfb154-b4c1-4952-8423-fa03f425a559\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:14.373324 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.373221 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60dca8f6-9229-4ddc-bf78-227ddd3950b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs\" (UID: \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\") " pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:14.373324 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.373267 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/60dca8f6-9229-4ddc-bf78-227ddd3950b4-kube-api-access-r89vf\") pod \"payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs\" (UID: \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\") " pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:14.374011 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.373983 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60dca8f6-9229-4ddc-bf78-227ddd3950b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs\" (UID: \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\") " pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:14.392416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.392381 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/60dca8f6-9229-4ddc-bf78-227ddd3950b4-kube-api-access-r89vf\") pod \"payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs\" (UID: \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\") " pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:14.407354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.407326 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:14.414828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.414772 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:14.464112 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.464072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:14.696705 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.696679 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228"] Nov 03 23:39:14.699020 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:14.698992 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbfb154_b4c1_4952_8423_fa03f425a559.slice/crio-7b85d0b84b4f7df6f2c26fc0fb30a308c02779989d8744cc3e7f9df2138ffa8a WatchSource:0}: Error finding container 7b85d0b84b4f7df6f2c26fc0fb30a308c02779989d8744cc3e7f9df2138ffa8a: Status 404 returned error can't find the container with id 7b85d0b84b4f7df6f2c26fc0fb30a308c02779989d8744cc3e7f9df2138ffa8a Nov 03 23:39:14.705408 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.705237 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs"] Nov 03 23:39:14.707243 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:14.707215 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60dca8f6_9229_4ddc_bf78_227ddd3950b4.slice/crio-12846b81c0d59ebdc1e1dc1823c04d3cf018c3c818f82112be6f8c6000936989 WatchSource:0}: Error finding container 12846b81c0d59ebdc1e1dc1823c04d3cf018c3c818f82112be6f8c6000936989: Status 404 returned error can't find the container with id 12846b81c0d59ebdc1e1dc1823c04d3cf018c3c818f82112be6f8c6000936989 Nov 03 23:39:14.807728 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.807697 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:14.880149 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.880107 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx"] Nov 03 23:39:14.883807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.883790 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:14.890406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.890384 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-80942cbb\"" Nov 03 23:39:14.897890 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.897865 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx"] Nov 03 23:39:14.925901 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.925865 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2"] Nov 03 23:39:14.929423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.929405 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:14.943713 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.943645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:14.945711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.945689 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2"] Nov 03 23:39:14.952905 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.952839 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-80942cbb-dockercfg-s7mvk\"" Nov 03 23:39:14.981679 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.981633 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4nh\" (UniqueName: \"kubernetes.io/projected/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-kube-api-access-mz4nh\") pod \"payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx\" (UID: \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:14.981874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:14.981844 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx\" (UID: \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:15.083118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.083072 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8ea5c7-8174-402e-af0a-cda38241ce95-plugins-config-volume\") pod \"payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2\" (UID: \"5c8ea5c7-8174-402e-af0a-cda38241ce95\") " pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:15.083118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.083120 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ggd\" (UniqueName: \"kubernetes.io/projected/5c8ea5c7-8174-402e-af0a-cda38241ce95-kube-api-access-x6ggd\") pod \"payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2\" (UID: \"5c8ea5c7-8174-402e-af0a-cda38241ce95\") " pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:15.083590 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.083232 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx\" (UID: \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:15.083590 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.083322 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4nh\" (UniqueName: \"kubernetes.io/projected/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-kube-api-access-mz4nh\") pod \"payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx\" (UID: \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:15.083921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.083902 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx\" (UID: \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:15.104894 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.104859 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4nh\" (UniqueName: \"kubernetes.io/projected/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-kube-api-access-mz4nh\") pod \"payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx\" (UID: \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:15.178741 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.178704 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" event={"ID":"60dca8f6-9229-4ddc-bf78-227ddd3950b4","Type":"ContainerStarted","Data":"0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539"} Nov 03 23:39:15.178973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.178765 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" event={"ID":"60dca8f6-9229-4ddc-bf78-227ddd3950b4","Type":"ContainerStarted","Data":"12846b81c0d59ebdc1e1dc1823c04d3cf018c3c818f82112be6f8c6000936989"} Nov 03 23:39:15.178973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.178832 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:15.180885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.180856 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" event={"ID":"abbfb154-b4c1-4952-8423-fa03f425a559","Type":"ContainerStarted","Data":"aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e"} Nov 03 23:39:15.181043 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.180911 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" event={"ID":"abbfb154-b4c1-4952-8423-fa03f425a559","Type":"ContainerStarted","Data":"7b85d0b84b4f7df6f2c26fc0fb30a308c02779989d8744cc3e7f9df2138ffa8a"} Nov 03 23:39:15.181043 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.180940 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:15.184483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.184451 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8ea5c7-8174-402e-af0a-cda38241ce95-plugins-config-volume\") pod \"payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2\" (UID: \"5c8ea5c7-8174-402e-af0a-cda38241ce95\") " pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:15.184603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.184514 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ggd\" (UniqueName: \"kubernetes.io/projected/5c8ea5c7-8174-402e-af0a-cda38241ce95-kube-api-access-x6ggd\") pod \"payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2\" (UID: \"5c8ea5c7-8174-402e-af0a-cda38241ce95\") " pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:15.185702 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.185679 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8ea5c7-8174-402e-af0a-cda38241ce95-plugins-config-volume\") pod \"payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2\" (UID: \"5c8ea5c7-8174-402e-af0a-cda38241ce95\") " pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:15.196624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.196603 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:15.199781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.199765 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ggd\" (UniqueName: \"kubernetes.io/projected/5c8ea5c7-8174-402e-af0a-cda38241ce95-kube-api-access-x6ggd\") pod \"payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2\" (UID: \"5c8ea5c7-8174-402e-af0a-cda38241ce95\") " pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:15.203331 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.203266 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" podStartSLOduration=1.20325524 podStartE2EDuration="1.20325524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:15.200662109 +0000 UTC m=+574.023258272" watchObservedRunningTime="2026-11-03 23:39:15.20325524 +0000 UTC m=+574.025851423" Nov 03 23:39:15.237725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.237658 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" podStartSLOduration=1.237643292 podStartE2EDuration="1.237643292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:15.237243999 +0000 UTC m=+574.059840165" watchObservedRunningTime="2026-11-03 23:39:15.237643292 +0000 UTC m=+574.060239454" Nov 03 23:39:15.242194 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.242166 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:15.377146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.377110 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:15.512304 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.512275 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx"] Nov 03 23:39:15.515350 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:15.515319 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f283b0_20cd_4bdc_8a33_285bf326d1bb.slice/crio-4924898df675858224566354d90dcb2341d93ba6657b098be43c6e58bacd9ea6 WatchSource:0}: Error finding container 4924898df675858224566354d90dcb2341d93ba6657b098be43c6e58bacd9ea6: Status 404 returned error can't find the container with id 4924898df675858224566354d90dcb2341d93ba6657b098be43c6e58bacd9ea6 Nov 03 23:39:15.548361 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:15.548325 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8ea5c7_8174_402e_af0a_cda38241ce95.slice/crio-bb9308f97b3121061840c5745a45b506eaa4563218f0c51a71ef7de6d875027b WatchSource:0}: Error finding container bb9308f97b3121061840c5745a45b506eaa4563218f0c51a71ef7de6d875027b: Status 404 returned error can't find the container with id bb9308f97b3121061840c5745a45b506eaa4563218f0c51a71ef7de6d875027b Nov 03 23:39:15.548915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.548888 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2"] Nov 03 23:39:15.796816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.796728 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:15.934475 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:15.934434 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:16.189189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:16.189085 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" event={"ID":"5c8ea5c7-8174-402e-af0a-cda38241ce95","Type":"ContainerStarted","Data":"594cf276908f6453e7bb74727e7bfae331607014364f65de94b477048abb41af"} Nov 03 23:39:16.189189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:16.189148 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" event={"ID":"5c8ea5c7-8174-402e-af0a-cda38241ce95","Type":"ContainerStarted","Data":"bb9308f97b3121061840c5745a45b506eaa4563218f0c51a71ef7de6d875027b"} Nov 03 23:39:16.189665 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:16.189199 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:16.190506 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:16.190480 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" event={"ID":"f6f283b0-20cd-4bdc-8a33-285bf326d1bb","Type":"ContainerStarted","Data":"46cc42864cf33d53e46741920444977f5cf45649a84e204ce1b05222e227f4ac"} Nov 03 23:39:16.190608 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:16.190530 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:16.190608 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:16.190550 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" event={"ID":"f6f283b0-20cd-4bdc-8a33-285bf326d1bb","Type":"ContainerStarted","Data":"4924898df675858224566354d90dcb2341d93ba6657b098be43c6e58bacd9ea6"} Nov 03 23:39:16.223574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:16.223520 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" podStartSLOduration=2.223507589 podStartE2EDuration="2.223507589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:16.221899846 +0000 UTC m=+575.044496009" watchObservedRunningTime="2026-11-03 23:39:16.223507589 +0000 UTC m=+575.046103751" Nov 03 23:39:16.539908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:16.539874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:17.036933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.036881 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:17.155579 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.155504 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" podStartSLOduration=3.155482797 podStartE2EDuration="3.155482797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:16.27732666 +0000 UTC m=+575.099922822" watchObservedRunningTime="2026-11-03 23:39:17.155482797 +0000 UTC m=+575.978078961" Nov 03 23:39:17.156441 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.156409 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd"] Nov 03 23:39:17.162567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.162532 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:17.207815 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.207784 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9e9be6c8\"" Nov 03 23:39:17.210247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.210216 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd"] Nov 03 23:39:17.282273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.282237 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598"] Nov 03 23:39:17.287123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.287044 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:17.291985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.291950 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:17.303301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.303273 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598"] Nov 03 23:39:17.306297 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.306258 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9e9be6c8-dockercfg-pdb99\"" Nov 03 23:39:17.310979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.310948 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29c3d84-f959-47f8-8bae-e69097b8ccdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd\" (UID: \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:17.311205 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.311188 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxhb4\" (UniqueName: \"kubernetes.io/projected/a29c3d84-f959-47f8-8bae-e69097b8ccdd-kube-api-access-pxhb4\") pod \"payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd\" (UID: \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:17.412006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.411970 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhgbf\" (UniqueName: \"kubernetes.io/projected/99b0b19e-75f6-47ba-b050-4c2055d5ec65-kube-api-access-nhgbf\") pod \"payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598\" (UID: \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\") " pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:17.412227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.412124 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b0b19e-75f6-47ba-b050-4c2055d5ec65-plugins-config-volume\") pod \"payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598\" (UID: \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\") " pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:17.412227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.412182 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxhb4\" (UniqueName: \"kubernetes.io/projected/a29c3d84-f959-47f8-8bae-e69097b8ccdd-kube-api-access-pxhb4\") pod \"payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd\" (UID: \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:17.412227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.412213 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29c3d84-f959-47f8-8bae-e69097b8ccdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd\" (UID: \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:17.412922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.412904 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29c3d84-f959-47f8-8bae-e69097b8ccdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd\" (UID: \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:17.441436 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.441404 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxhb4\" (UniqueName: \"kubernetes.io/projected/a29c3d84-f959-47f8-8bae-e69097b8ccdd-kube-api-access-pxhb4\") pod \"payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd\" (UID: \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:17.482390 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.482346 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:17.513367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.513328 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b0b19e-75f6-47ba-b050-4c2055d5ec65-plugins-config-volume\") pod \"payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598\" (UID: \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\") " pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:17.513529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.513469 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhgbf\" (UniqueName: \"kubernetes.io/projected/99b0b19e-75f6-47ba-b050-4c2055d5ec65-kube-api-access-nhgbf\") pod \"payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598\" (UID: \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\") " pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:17.514319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.514295 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b0b19e-75f6-47ba-b050-4c2055d5ec65-plugins-config-volume\") pod \"payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598\" (UID: \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\") " pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:17.529045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.528989 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhgbf\" (UniqueName: \"kubernetes.io/projected/99b0b19e-75f6-47ba-b050-4c2055d5ec65-kube-api-access-nhgbf\") pod \"payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598\" (UID: \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\") " pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:17.605206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.605169 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:17.777096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.776396 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd"] Nov 03 23:39:17.777413 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:17.777386 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29c3d84_f959_47f8_8bae_e69097b8ccdd.slice/crio-be92381f61211e039f749d167fa9f88f4d2aefe8c936d375f7e626b8fad08eee WatchSource:0}: Error finding container be92381f61211e039f749d167fa9f88f4d2aefe8c936d375f7e626b8fad08eee: Status 404 returned error can't find the container with id be92381f61211e039f749d167fa9f88f4d2aefe8c936d375f7e626b8fad08eee Nov 03 23:39:17.925638 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.925608 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598"] Nov 03 23:39:17.927766 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:17.927739 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b0b19e_75f6_47ba_b050_4c2055d5ec65.slice/crio-be5f8849f76b10b0c15c442eb4b4c47999bf48e4fcbc7806fdf97db6433dd45e WatchSource:0}: Error finding container be5f8849f76b10b0c15c442eb4b4c47999bf48e4fcbc7806fdf97db6433dd45e: Status 404 returned error can't find the container with id be5f8849f76b10b0c15c442eb4b4c47999bf48e4fcbc7806fdf97db6433dd45e Nov 03 23:39:17.977808 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:17.977784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:18.206256 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:18.206218 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" event={"ID":"99b0b19e-75f6-47ba-b050-4c2055d5ec65","Type":"ContainerStarted","Data":"1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3"} Nov 03 23:39:18.206430 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:18.206282 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" event={"ID":"99b0b19e-75f6-47ba-b050-4c2055d5ec65","Type":"ContainerStarted","Data":"be5f8849f76b10b0c15c442eb4b4c47999bf48e4fcbc7806fdf97db6433dd45e"} Nov 03 23:39:18.206505 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:18.206439 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:18.207964 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:18.207940 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" event={"ID":"a29c3d84-f959-47f8-8bae-e69097b8ccdd","Type":"ContainerStarted","Data":"4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97"} Nov 03 23:39:18.208386 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:18.207976 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" event={"ID":"a29c3d84-f959-47f8-8bae-e69097b8ccdd","Type":"ContainerStarted","Data":"be92381f61211e039f749d167fa9f88f4d2aefe8c936d375f7e626b8fad08eee"} Nov 03 23:39:18.208386 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:18.208023 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:18.276721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:18.276622 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" podStartSLOduration=1.276599447 podStartE2EDuration="1.276599447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:18.229214651 +0000 UTC m=+577.051810816" watchObservedRunningTime="2026-11-03 23:39:18.276599447 +0000 UTC m=+577.099195611" Nov 03 23:39:18.315252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:18.315218 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:19.709971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:19.709938 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:20.040082 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:20.039976 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:20.298565 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:20.298470 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:20.998085 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:20.998050 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:21.169069 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:21.167461 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:21.407265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:21.407179 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:21.627381 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:21.627352 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:21.810345 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:21.810306 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:22.327887 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.327811 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" podStartSLOduration=5.327793546 podStartE2EDuration="5.327793546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:18.277671709 +0000 UTC m=+577.100267871" watchObservedRunningTime="2026-11-03 23:39:22.327793546 +0000 UTC m=+581.150389709" Nov 03 23:39:22.331364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.331333 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws"] Nov 03 23:39:22.335881 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.335858 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.349203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.349182 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-044f4dcd-openshift-default-dockercfg-vpgfk\"" Nov 03 23:39:22.377055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.370786 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws"] Nov 03 23:39:22.392701 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.392669 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-credential-socket\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.392874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.392713 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zvr\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-kube-api-access-h8zvr\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.392874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.392808 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-socket\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.392874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.392846 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-envoy\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.392874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.392869 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-token\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.393018 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.392990 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-certs\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.393018 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.393013 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-podinfo\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.393119 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.393061 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-data\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.393185 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.393171 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc4913-62bd-41d6-be48-858dc92d8918-istiod-ca-cert\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.484060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.483933 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.493752 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zvr\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-kube-api-access-h8zvr\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.493797 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-socket\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.493822 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-envoy\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.493865 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-token\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.493901 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-certs\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.493919 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-podinfo\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.493948 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-data\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.494023 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc4913-62bd-41d6-be48-858dc92d8918-istiod-ca-cert\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.494102 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-credential-socket\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.494676 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-credential-socket\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.494949 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-socket\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.495400 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-certs\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.495800 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-data\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.496614 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc4913-62bd-41d6-be48-858dc92d8918-istiod-ca-cert\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.507057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.500000 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-envoy\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.514239 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.514198 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-podinfo\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.517563 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.517528 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-token\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.526978 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.526921 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zvr\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-kube-api-access-h8zvr\") pod \"e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.652190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.652112 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:22.820161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:22.820121 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:23.009082 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.007167 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm"] Nov 03 23:39:23.012260 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.012236 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.020277 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.020228 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:39:23.020340 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.020325 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:39:23.029628 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.029598 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm"] Nov 03 23:39:23.035026 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.034995 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws"] Nov 03 23:39:23.040060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.039554 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:23.040060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.039627 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:23.040060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.039671 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:23.043871 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.043658 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9e9be6c8-serving-cert\"" Nov 03 23:39:23.043871 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.043778 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:39:23.043871 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.043848 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xflt\"" Nov 03 23:39:23.109865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.109830 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-maas-api-tls\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.110114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.109902 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvl6w\" (UniqueName: \"kubernetes.io/projected/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-kube-api-access-dvl6w\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.110114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.109966 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-openshift-service-ca\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.110114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.110073 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-metrics-certs\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.148755 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.148725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:23.211265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.211235 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvl6w\" (UniqueName: \"kubernetes.io/projected/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-kube-api-access-dvl6w\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.211412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.211294 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-openshift-service-ca\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.211412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.211329 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-metrics-certs\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.211524 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.211504 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-maas-api-tls\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.212348 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.212324 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-openshift-service-ca\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.213719 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.213701 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-metrics-certs\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.214066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.214048 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-maas-api-tls\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.230737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.230706 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvl6w\" (UniqueName: \"kubernetes.io/projected/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-kube-api-access-dvl6w\") pod \"maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.246823 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.246710 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" event={"ID":"4ddc4913-62bd-41d6-be48-858dc92d8918","Type":"ContainerStarted","Data":"6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339"} Nov 03 23:39:23.246823 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.246759 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" event={"ID":"4ddc4913-62bd-41d6-be48-858dc92d8918","Type":"ContainerStarted","Data":"52926d1002e98a11a9f8225ee857139981145c5dd684bd4686b1b4f6aade99c9"} Nov 03 23:39:23.330091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.326111 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:23.368885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.368705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:23.652840 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.652735 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:23.718604 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.718537 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" podStartSLOduration=1.718519082 podStartE2EDuration="1.718519082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:23.319965331 +0000 UTC m=+582.142561511" watchObservedRunningTime="2026-11-03 23:39:23.718519082 +0000 UTC m=+582.541115243" Nov 03 23:39:23.719148 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:23.719126 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm"] Nov 03 23:39:23.728846 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:23.728080 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a320c23_b1f3_47f7_bbae_8198d1a0ce1c.slice/crio-8da05ddd842c24dac25a4d506b8a2de0989c08ab8798bf11d6945eabfd7d96fc WatchSource:0}: Error finding container 8da05ddd842c24dac25a4d506b8a2de0989c08ab8798bf11d6945eabfd7d96fc: Status 404 returned error can't find the container with id 8da05ddd842c24dac25a4d506b8a2de0989c08ab8798bf11d6945eabfd7d96fc Nov 03 23:39:24.257058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:24.253907 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" event={"ID":"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c","Type":"ContainerStarted","Data":"8da05ddd842c24dac25a4d506b8a2de0989c08ab8798bf11d6945eabfd7d96fc"} Nov 03 23:39:24.448970 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:24.448931 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:24.682703 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:24.682333 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:24.778081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:24.777145 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:24.892325 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:24.892251 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:25.283551 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:25.283514 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" event={"ID":"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c","Type":"ContainerStarted","Data":"fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1"} Nov 03 23:39:25.283820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:25.283800 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:25.283934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:25.283849 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:25.284879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:25.284854 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:25.308183 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:25.308133 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" podStartSLOduration=2.6629334350000002 podStartE2EDuration="3.308120249s" totalImagesPullingTime="645.186814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:25.306423358 +0000 UTC m=+584.129019543" watchObservedRunningTime="2026-11-03 23:39:25.308120249 +0000 UTC m=+584.130716455" Nov 03 23:39:25.551704 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:25.551614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:26.110261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:26.110118 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:26.195315 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:26.195231 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:26.196695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:26.196619 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:26.414522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:26.414395 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:26.985438 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:26.985401 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:27.200216 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:27.200188 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:27.200390 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:27.200251 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:28.157381 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:28.157342 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:28.211244 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:28.211199 6208 patch_prober.go:29] interesting pod/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:39:28.211407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:28.211272 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:39:29.223737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.223703 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:29.224260 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.223789 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:29.324960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.324927 6208 generic.go:356] "Generic (PLEG): container finished" podID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerID="394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8" exitCode=0 Nov 03 23:39:29.325175 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.324979 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" event={"ID":"dfa30bbe-ebf1-4dba-b206-41223aa75c1d","Type":"ContainerDied","Data":"394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8"} Nov 03 23:39:29.325369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.325348 6208 scope.go:122] "RemoveContainer" containerID="394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8" Nov 03 23:39:29.326853 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.326818 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:29.326957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.326909 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:29.327019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.326974 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:29.364847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.364815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:29.501781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:29.501699 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:30.332087 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:30.332053 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" event={"ID":"dfa30bbe-ebf1-4dba-b206-41223aa75c1d","Type":"ContainerStarted","Data":"f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62"} Nov 03 23:39:30.359718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:30.359616 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:30.566536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:30.566502 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:30.566899 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:30.566875 6208 patch_prober.go:29] interesting pod/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 23:39:30.567079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:30.567056 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 23:39:31.298476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:31.298445 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:31.336199 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:31.336172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:31.578914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:31.578830 6208 patch_prober.go:29] interesting pod/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 23:39:31.578914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:31.578889 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 23:39:32.345535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:32.345499 6208 generic.go:356] "Generic (PLEG): container finished" podID="908dd01e-6450-44ea-987d-989cf05595a4" containerID="343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba" exitCode=0 Nov 03 23:39:32.345990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:32.345554 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" event={"ID":"908dd01e-6450-44ea-987d-989cf05595a4","Type":"ContainerDied","Data":"343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba"} Nov 03 23:39:32.345990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:32.345954 6208 scope.go:122] "RemoveContainer" containerID="343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba" Nov 03 23:39:32.347371 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:32.347337 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:32.347467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:32.347409 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:32.347467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:32.347441 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:32.581137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:32.581102 6208 patch_prober.go:29] interesting pod/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 23:39:32.581305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:32.581161 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 23:39:32.688197 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:32.688099 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:33.375058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:33.367260 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" event={"ID":"908dd01e-6450-44ea-987d-989cf05595a4","Type":"ContainerStarted","Data":"653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c"} Nov 03 23:39:33.446232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:33.446200 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:33.446451 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:33.446424 6208 patch_prober.go:29] interesting pod/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 23:39:33.446510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:33.446490 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 23:39:33.520057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:33.519198 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:33.569059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:33.567134 6208 patch_prober.go:29] interesting pod/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 23:39:33.569059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:33.567208 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 23:39:33.746072 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:33.743604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:34.078113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:34.077694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:34.446594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:34.446515 6208 patch_prober.go:29] interesting pod/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 23:39:34.446971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:34.446593 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 23:39:34.576059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:34.573225 6208 patch_prober.go:29] interesting pod/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 23:39:34.576059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:34.573285 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 23:39:34.652200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:34.652166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:34.980061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:34.976127 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt"] Nov 03 23:39:35.020057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.019559 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:35.033054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.031448 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt"] Nov 03 23:39:35.082058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.077384 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d3157179\"" Nov 03 23:39:35.082058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.077631 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d3157179-dockercfg-8qn7l\"" Nov 03 23:39:35.137445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.137403 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:35.149251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.149216 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-kube-api-access-h929s\") pod \"payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt\" (UID: \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:35.149396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.149289 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt\" (UID: \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:35.250303 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.250211 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-kube-api-access-h929s\") pod \"payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt\" (UID: \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:35.250303 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.250275 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt\" (UID: \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:35.251151 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.251131 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt\" (UID: \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:35.298491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.297330 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-kube-api-access-h929s\") pod \"payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt\" (UID: \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\") " pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:35.342500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.342463 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:35.447123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.447090 6208 patch_prober.go:29] interesting pod/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 23:39:35.447481 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.447157 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 23:39:35.575077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.567593 6208 patch_prober.go:29] interesting pod/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 03 23:39:35.575077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.567654 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 03 23:39:35.744440 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:35.744381 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:39:35.804324 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:35.804222 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt"] Nov 03 23:39:35.818247 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:35.818215 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb5ccde_6895_44ef_a0cc_d40d2ef42ae1.slice/crio-7a81af34ff4e2633821dc0313d320e2e2e66f94bc90eef82221b756ad6809fc2 WatchSource:0}: Error finding container 7a81af34ff4e2633821dc0313d320e2e2e66f94bc90eef82221b756ad6809fc2: Status 404 returned error can't find the container with id 7a81af34ff4e2633821dc0313d320e2e2e66f94bc90eef82221b756ad6809fc2 Nov 03 23:39:36.047270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.047232 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:36.447613 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.447345 6208 patch_prober.go:29] interesting pod/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 23:39:36.447613 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.447397 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 23:39:36.472056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.466964 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" event={"ID":"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1","Type":"ContainerStarted","Data":"3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04"} Nov 03 23:39:36.472056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.467047 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" event={"ID":"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1","Type":"ContainerStarted","Data":"7a81af34ff4e2633821dc0313d320e2e2e66f94bc90eef82221b756ad6809fc2"} Nov 03 23:39:36.472056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.467902 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:36.595597 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.595554 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:36.597894 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.597873 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:36.602453 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.602431 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:36.629169 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:36.629119 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" podStartSLOduration=2.629106118 podStartE2EDuration="2.629106118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:36.504339375 +0000 UTC m=+595.326935547" watchObservedRunningTime="2026-11-03 23:39:36.629106118 +0000 UTC m=+595.451702280" Nov 03 23:39:37.446989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:37.446955 6208 patch_prober.go:29] interesting pod/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 23:39:37.447195 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:37.447006 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 23:39:37.505537 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:37.505484 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:38.250776 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:38.250730 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:38.447255 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:38.447224 6208 patch_prober.go:29] interesting pod/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 23:39:38.447412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:38.447287 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 23:39:39.140695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:39.140660 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:39.240780 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:39.240705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:39.342846 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:39.342817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:39.451929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:39.451900 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:39.452277 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:39.452253 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:39.452963 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:39.452946 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:40.151195 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:40.151149 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:41.045971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:41.045934 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:41.325311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:41.325214 6208 patch_prober.go:29] interesting pod/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:39:41.325311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:41.325275 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:39:41.352937 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:41.352907 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:39:41.353146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:41.352959 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:39:42.255777 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:42.255705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:42.532264 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:42.532230 6208 generic.go:356] "Generic (PLEG): container finished" podID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerID="6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339" exitCode=0 Nov 03 23:39:42.532698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:42.532308 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" event={"ID":"4ddc4913-62bd-41d6-be48-858dc92d8918","Type":"ContainerDied","Data":"6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339"} Nov 03 23:39:42.532850 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:42.532830 6208 scope.go:122] "RemoveContainer" containerID="6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339" Nov 03 23:39:42.534241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:42.534214 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:42.534318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:42.534304 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:42.534357 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:42.534340 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:43.285843 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:43.285797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:43.540101 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:43.540009 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" event={"ID":"4ddc4913-62bd-41d6-be48-858dc92d8918","Type":"ContainerStarted","Data":"981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f"} Nov 03 23:39:43.653452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:43.653416 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:43.653788 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:43.653768 6208 patch_prober.go:29] interesting pod/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 23:39:43.661171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:43.661120 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 23:39:44.151059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:44.150994 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:44.391612 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:44.391574 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2"] Nov 03 23:39:44.409492 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:44.409397 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx"] Nov 03 23:39:44.481223 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:44.481193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:44.680201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:44.678158 6208 patch_prober.go:29] interesting pod/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 23:39:44.680201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:44.678225 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 23:39:44.705865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:44.705831 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:45.048726 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:45.048690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:45.274056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:45.270949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:45.653207 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:45.653168 6208 patch_prober.go:29] interesting pod/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 23:39:45.653516 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:45.653485 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 23:39:45.749479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:45.749440 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:46.652474 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:46.652439 6208 patch_prober.go:29] interesting pod/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 23:39:46.652671 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:46.652492 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 23:39:46.676161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:46.676125 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:47.642820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:47.642783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:47.664060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:47.663428 6208 patch_prober.go:29] interesting pod/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 23:39:47.664060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:47.663492 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 23:39:47.932609 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:47.932525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:48.377058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:48.376904 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:48.512744 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:48.512709 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:39:48.651866 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:48.651773 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs"] Nov 03 23:39:48.673839 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:48.673799 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228"] Nov 03 23:39:48.893481 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:48.893434 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:48.921778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:48.921654 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:48.922583 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:48.922561 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:48.926196 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:48.926169 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:39:49.038372 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.038341 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:49.392204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.392159 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" podUID="5c8ea5c7-8174-402e-af0a-cda38241ce95" containerName="payload-processing" containerID="cri-o://594cf276908f6453e7bb74727e7bfae331607014364f65de94b477048abb41af" gracePeriod=25 Nov 03 23:39:49.409840 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.409810 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" podUID="f6f283b0-20cd-4bdc-8a33-285bf326d1bb" containerName="payload-pre-processing" containerID="cri-o://46cc42864cf33d53e46741920444977f5cf45649a84e204ce1b05222e227f4ac" gracePeriod=25 Nov 03 23:39:49.607227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.607198 6208 generic.go:356] "Generic (PLEG): container finished" podID="5c8ea5c7-8174-402e-af0a-cda38241ce95" containerID="594cf276908f6453e7bb74727e7bfae331607014364f65de94b477048abb41af" exitCode=0 Nov 03 23:39:49.607378 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.607318 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" event={"ID":"5c8ea5c7-8174-402e-af0a-cda38241ce95","Type":"ContainerDied","Data":"594cf276908f6453e7bb74727e7bfae331607014364f65de94b477048abb41af"} Nov 03 23:39:49.608778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.608753 6208 generic.go:356] "Generic (PLEG): container finished" podID="f6f283b0-20cd-4bdc-8a33-285bf326d1bb" containerID="46cc42864cf33d53e46741920444977f5cf45649a84e204ce1b05222e227f4ac" exitCode=0 Nov 03 23:39:49.609347 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.609329 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" event={"ID":"f6f283b0-20cd-4bdc-8a33-285bf326d1bb","Type":"ContainerDied","Data":"46cc42864cf33d53e46741920444977f5cf45649a84e204ce1b05222e227f4ac"} Nov 03 23:39:49.669842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.669812 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:49.727651 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.727621 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:49.806493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.806462 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-kube-api-access-mz4nh\" (UniqueName: \"kubernetes.io/projected/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-kube-api-access-mz4nh\") pod \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\" (UID: \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\") " Nov 03 23:39:49.806693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.806511 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c8ea5c7-8174-402e-af0a-cda38241ce95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8ea5c7-8174-402e-af0a-cda38241ce95-plugins-config-volume\") pod \"5c8ea5c7-8174-402e-af0a-cda38241ce95\" (UID: \"5c8ea5c7-8174-402e-af0a-cda38241ce95\") " Nov 03 23:39:49.806693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.806535 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8ea5c7-8174-402e-af0a-cda38241ce95-kube-api-access-x6ggd\" (UniqueName: \"kubernetes.io/projected/5c8ea5c7-8174-402e-af0a-cda38241ce95-kube-api-access-x6ggd\") pod \"5c8ea5c7-8174-402e-af0a-cda38241ce95\" (UID: \"5c8ea5c7-8174-402e-af0a-cda38241ce95\") " Nov 03 23:39:49.806693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.806583 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-plugins-config-volume\") pod \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\" (UID: \"f6f283b0-20cd-4bdc-8a33-285bf326d1bb\") " Nov 03 23:39:49.806934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.806908 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8ea5c7-8174-402e-af0a-cda38241ce95-plugins-config-volume" pod "5c8ea5c7-8174-402e-af0a-cda38241ce95" (UID: "5c8ea5c7-8174-402e-af0a-cda38241ce95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:39:49.807013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.806995 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-plugins-config-volume" pod "f6f283b0-20cd-4bdc-8a33-285bf326d1bb" (UID: "f6f283b0-20cd-4bdc-8a33-285bf326d1bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:39:49.808811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.808780 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8ea5c7-8174-402e-af0a-cda38241ce95-kube-api-access-x6ggd" pod "5c8ea5c7-8174-402e-af0a-cda38241ce95" (UID: "5c8ea5c7-8174-402e-af0a-cda38241ce95"). InnerVolumeSpecName "kube-api-access-x6ggd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:49.808913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.808875 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-kube-api-access-mz4nh" pod "f6f283b0-20cd-4bdc-8a33-285bf326d1bb" (UID: "f6f283b0-20cd-4bdc-8a33-285bf326d1bb"). InnerVolumeSpecName "kube-api-access-mz4nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:49.907700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.907617 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8ea5c7-8174-402e-af0a-cda38241ce95-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:49.907700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.907650 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ggd\" (UniqueName: \"kubernetes.io/projected/5c8ea5c7-8174-402e-af0a-cda38241ce95-kube-api-access-x6ggd\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:49.907700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.907660 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:49.907700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:49.907670 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4nh\" (UniqueName: \"kubernetes.io/projected/f6f283b0-20cd-4bdc-8a33-285bf326d1bb-kube-api-access-mz4nh\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:50.100300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.100268 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:50.249534 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.249491 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:50.615958 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.615862 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" Nov 03 23:39:50.615958 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.615895 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2" event={"ID":"5c8ea5c7-8174-402e-af0a-cda38241ce95","Type":"ContainerDied","Data":"bb9308f97b3121061840c5745a45b506eaa4563218f0c51a71ef7de6d875027b"} Nov 03 23:39:50.616188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.615966 6208 scope.go:122] "RemoveContainer" containerID="594cf276908f6453e7bb74727e7bfae331607014364f65de94b477048abb41af" Nov 03 23:39:50.617428 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.617408 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" Nov 03 23:39:50.617494 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.617412 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx" event={"ID":"f6f283b0-20cd-4bdc-8a33-285bf326d1bb","Type":"ContainerDied","Data":"4924898df675858224566354d90dcb2341d93ba6657b098be43c6e58bacd9ea6"} Nov 03 23:39:50.628100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.628077 6208 scope.go:122] "RemoveContainer" containerID="46cc42864cf33d53e46741920444977f5cf45649a84e204ce1b05222e227f4ac" Nov 03 23:39:50.647580 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.647553 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:50.656097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.656067 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv"] Nov 03 23:39:50.656325 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.656303 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" containerID="cri-o://653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c" gracePeriod=30 Nov 03 23:39:50.664785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.664749 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2"] Nov 03 23:39:50.666944 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.666924 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-80942cbb-7f89cd6c66-hknq2"] Nov 03 23:39:50.716773 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.716740 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx"] Nov 03 23:39:50.723815 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:50.723786 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-80942cbb-77759645c5-kbctx"] Nov 03 23:39:51.004793 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.004760 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:51.293839 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.293758 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c8ea5c7-8174-402e-af0a-cda38241ce95" path="/var/lib/kubelet/pods/5c8ea5c7-8174-402e-af0a-cda38241ce95/volumes" Nov 03 23:39:51.294189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.294174 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f283b0-20cd-4bdc-8a33-285bf326d1bb" path="/var/lib/kubelet/pods/f6f283b0-20cd-4bdc-8a33-285bf326d1bb/volumes" Nov 03 23:39:51.299247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.299221 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:51.527265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.527234 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:51.768394 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.768347 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt"] Nov 03 23:39:51.768892 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.768644 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" containerID="cri-o://f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62" gracePeriod=30 Nov 03 23:39:51.772499 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.772469 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:51.780025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.780002 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598"] Nov 03 23:39:51.851488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:51.851450 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd"] Nov 03 23:39:52.043106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:52.042986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:52.227816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:52.227787 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:52.341027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:52.340923 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:52.851961 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:52.851925 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:53.652569 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:53.652528 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" podUID="60dca8f6-9229-4ddc-bf78-227ddd3950b4" containerName="payload-processing" containerID="cri-o://0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539" gracePeriod=25 Nov 03 23:39:53.674204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:53.674172 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" podUID="abbfb154-b4c1-4952-8423-fa03f425a559" containerName="payload-pre-processing" containerID="cri-o://aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e" gracePeriod=25 Nov 03 23:39:53.780443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:53.780397 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:53.929719 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:53.929695 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:53.966430 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:53.966403 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:54.050160 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.050128 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60dca8f6-9229-4ddc-bf78-227ddd3950b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60dca8f6-9229-4ddc-bf78-227ddd3950b4-plugins-config-volume\") pod \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\" (UID: \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\") " Nov 03 23:39:54.050343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.050173 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbfb154-b4c1-4952-8423-fa03f425a559-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbfb154-b4c1-4952-8423-fa03f425a559-plugins-config-volume\") pod \"abbfb154-b4c1-4952-8423-fa03f425a559\" (UID: \"abbfb154-b4c1-4952-8423-fa03f425a559\") " Nov 03 23:39:54.050343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.050227 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbfb154-b4c1-4952-8423-fa03f425a559-kube-api-access-g5crb\" (UniqueName: \"kubernetes.io/projected/abbfb154-b4c1-4952-8423-fa03f425a559-kube-api-access-g5crb\") pod \"abbfb154-b4c1-4952-8423-fa03f425a559\" (UID: \"abbfb154-b4c1-4952-8423-fa03f425a559\") " Nov 03 23:39:54.050343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.050292 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60dca8f6-9229-4ddc-bf78-227ddd3950b4-kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/60dca8f6-9229-4ddc-bf78-227ddd3950b4-kube-api-access-r89vf\") pod \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\" (UID: \"60dca8f6-9229-4ddc-bf78-227ddd3950b4\") " Nov 03 23:39:54.050548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.050523 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60dca8f6-9229-4ddc-bf78-227ddd3950b4-plugins-config-volume" pod "60dca8f6-9229-4ddc-bf78-227ddd3950b4" (UID: "60dca8f6-9229-4ddc-bf78-227ddd3950b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:39:54.050636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.050613 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbfb154-b4c1-4952-8423-fa03f425a559-plugins-config-volume" pod "abbfb154-b4c1-4952-8423-fa03f425a559" (UID: "abbfb154-b4c1-4952-8423-fa03f425a559"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:39:54.050702 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.050678 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60dca8f6-9229-4ddc-bf78-227ddd3950b4-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:54.052415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.052391 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60dca8f6-9229-4ddc-bf78-227ddd3950b4-kube-api-access-r89vf" pod "60dca8f6-9229-4ddc-bf78-227ddd3950b4" (UID: "60dca8f6-9229-4ddc-bf78-227ddd3950b4"). InnerVolumeSpecName "kube-api-access-r89vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:54.052508 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.052441 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbfb154-b4c1-4952-8423-fa03f425a559-kube-api-access-g5crb" pod "abbfb154-b4c1-4952-8423-fa03f425a559" (UID: "abbfb154-b4c1-4952-8423-fa03f425a559"). InnerVolumeSpecName "kube-api-access-g5crb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:54.151756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.151717 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5crb\" (UniqueName: \"kubernetes.io/projected/abbfb154-b4c1-4952-8423-fa03f425a559-kube-api-access-g5crb\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:54.151756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.151751 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/60dca8f6-9229-4ddc-bf78-227ddd3950b4-kube-api-access-r89vf\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:54.151951 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.151765 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abbfb154-b4c1-4952-8423-fa03f425a559-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:54.454332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.454293 6208 patch_prober.go:29] interesting pod/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:39:54.454532 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.454358 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:39:54.650176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.650140 6208 generic.go:356] "Generic (PLEG): container finished" podID="abbfb154-b4c1-4952-8423-fa03f425a559" containerID="aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e" exitCode=0 Nov 03 23:39:54.650369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.650192 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" Nov 03 23:39:54.650369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.650234 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" event={"ID":"abbfb154-b4c1-4952-8423-fa03f425a559","Type":"ContainerDied","Data":"aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e"} Nov 03 23:39:54.650369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.650294 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228" event={"ID":"abbfb154-b4c1-4952-8423-fa03f425a559","Type":"ContainerDied","Data":"7b85d0b84b4f7df6f2c26fc0fb30a308c02779989d8744cc3e7f9df2138ffa8a"} Nov 03 23:39:54.650369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.650316 6208 scope.go:122] "RemoveContainer" containerID="aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e" Nov 03 23:39:54.651694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.651676 6208 generic.go:356] "Generic (PLEG): container finished" podID="60dca8f6-9229-4ddc-bf78-227ddd3950b4" containerID="0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539" exitCode=0 Nov 03 23:39:54.651797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.651705 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" event={"ID":"60dca8f6-9229-4ddc-bf78-227ddd3950b4","Type":"ContainerDied","Data":"0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539"} Nov 03 23:39:54.651797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.651736 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" Nov 03 23:39:54.651797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.651752 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs" event={"ID":"60dca8f6-9229-4ddc-bf78-227ddd3950b4","Type":"ContainerDied","Data":"12846b81c0d59ebdc1e1dc1823c04d3cf018c3c818f82112be6f8c6000936989"} Nov 03 23:39:54.661517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.661328 6208 scope.go:122] "RemoveContainer" containerID="aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e" Nov 03 23:39:54.661583 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:54.661567 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e\": container with ID starting with aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e not found: ID does not exist" containerID="aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e" Nov 03 23:39:54.661619 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.661591 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e"} err="failed to get container status \"aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e\": rpc error: code = NotFound desc = could not find container \"aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e\": container with ID starting with aaef651e9c1fdee0d1800defe0701447ff06f922bf81d9cf696bf684df8cd61e not found: ID does not exist" Nov 03 23:39:54.661619 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.661606 6208 scope.go:122] "RemoveContainer" containerID="0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539" Nov 03 23:39:54.681382 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.681363 6208 scope.go:122] "RemoveContainer" containerID="0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539" Nov 03 23:39:54.681680 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:54.681661 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539\": container with ID starting with 0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539 not found: ID does not exist" containerID="0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539" Nov 03 23:39:54.681733 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.681689 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539"} err="failed to get container status \"0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539\": rpc error: code = NotFound desc = could not find container \"0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539\": container with ID starting with 0b966d1f2200ee4c5956616c54d9f294dc81822948a4938b5dd28ca367a02539 not found: ID does not exist" Nov 03 23:39:54.694767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.694744 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228"] Nov 03 23:39:54.700341 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.700313 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7c2a2d28-c665cc8f9-mc228"] Nov 03 23:39:54.718518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.718493 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs"] Nov 03 23:39:54.737362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.737334 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7c2a2d28-8476f58bb4-qhsfs"] Nov 03 23:39:54.995749 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:54.995656 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:55.293388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:55.293314 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60dca8f6-9229-4ddc-bf78-227ddd3950b4" path="/var/lib/kubelet/pods/60dca8f6-9229-4ddc-bf78-227ddd3950b4/volumes" Nov 03 23:39:55.293699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:55.293686 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbfb154-b4c1-4952-8423-fa03f425a559" path="/var/lib/kubelet/pods/abbfb154-b4c1-4952-8423-fa03f425a559/volumes" Nov 03 23:39:55.333215 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:55.333176 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:55.479478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:55.479438 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:55.939731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:55.939707 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:56.072312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072280 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908dd01e-6450-44ea-987d-989cf05595a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd01e-6450-44ea-987d-989cf05595a4-istiod-ca-cert\") pod \"908dd01e-6450-44ea-987d-989cf05595a4\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " Nov 03 23:39:56.072825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072330 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-kube-api-access-4bttw\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-kube-api-access-4bttw\") pod \"908dd01e-6450-44ea-987d-989cf05595a4\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " Nov 03 23:39:56.072825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072376 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-envoy\") pod \"908dd01e-6450-44ea-987d-989cf05595a4\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " Nov 03 23:39:56.072825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072396 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/908dd01e-6450-44ea-987d-989cf05595a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd01e-6450-44ea-987d-989cf05595a4-istio-podinfo\") pod \"908dd01e-6450-44ea-987d-989cf05595a4\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " Nov 03 23:39:56.072825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072420 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-istio-token\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-istio-token\") pod \"908dd01e-6450-44ea-987d-989cf05595a4\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " Nov 03 23:39:56.072825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072501 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-data\") pod \"908dd01e-6450-44ea-987d-989cf05595a4\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " Nov 03 23:39:56.072825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072545 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-credential-socket\") pod \"908dd01e-6450-44ea-987d-989cf05595a4\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " Nov 03 23:39:56.072825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072571 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-socket\") pod \"908dd01e-6450-44ea-987d-989cf05595a4\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " Nov 03 23:39:56.072825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072611 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-certs\") pod \"908dd01e-6450-44ea-987d-989cf05595a4\" (UID: \"908dd01e-6450-44ea-987d-989cf05595a4\") " Nov 03 23:39:56.072825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072775 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908dd01e-6450-44ea-987d-989cf05595a4-istiod-ca-cert" pod "908dd01e-6450-44ea-987d-989cf05595a4" (UID: "908dd01e-6450-44ea-987d-989cf05595a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:39:56.073332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072887 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-credential-socket" pod "908dd01e-6450-44ea-987d-989cf05595a4" (UID: "908dd01e-6450-44ea-987d-989cf05595a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:56.073332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.072898 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-data" pod "908dd01e-6450-44ea-987d-989cf05595a4" (UID: "908dd01e-6450-44ea-987d-989cf05595a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:56.073332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.073025 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-certs" pod "908dd01e-6450-44ea-987d-989cf05595a4" (UID: "908dd01e-6450-44ea-987d-989cf05595a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:56.073332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.073117 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd01e-6450-44ea-987d-989cf05595a4-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.073332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.073142 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.073332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.073158 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.073332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.073157 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-socket" pod "908dd01e-6450-44ea-987d-989cf05595a4" (UID: "908dd01e-6450-44ea-987d-989cf05595a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:56.075019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.074984 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-envoy" pod "908dd01e-6450-44ea-987d-989cf05595a4" (UID: "908dd01e-6450-44ea-987d-989cf05595a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:56.076477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.076444 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/908dd01e-6450-44ea-987d-989cf05595a4-istio-podinfo" pod "908dd01e-6450-44ea-987d-989cf05595a4" (UID: "908dd01e-6450-44ea-987d-989cf05595a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:39:56.076477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.076466 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-istio-token" pod "908dd01e-6450-44ea-987d-989cf05595a4" (UID: "908dd01e-6450-44ea-987d-989cf05595a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:56.076658 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.076518 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-kube-api-access-4bttw" pod "908dd01e-6450-44ea-987d-989cf05595a4" (UID: "908dd01e-6450-44ea-987d-989cf05595a4"). InnerVolumeSpecName "kube-api-access-4bttw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:56.109887 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.109852 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm"] Nov 03 23:39:56.110128 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.110106 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" podUID="6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" containerName="maas-api" containerID="cri-o://fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1" gracePeriod=30 Nov 03 23:39:56.174003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.173926 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.174003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.173960 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd01e-6450-44ea-987d-989cf05595a4-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.174003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.173971 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.174003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.173980 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.174003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.173990 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd01e-6450-44ea-987d-989cf05595a4-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.174003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.173999 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bttw\" (UniqueName: \"kubernetes.io/projected/908dd01e-6450-44ea-987d-989cf05595a4-kube-api-access-4bttw\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.182280 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.182249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:56.361897 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.361861 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:56.476372 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.476336 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-openshift-service-ca\") pod \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " Nov 03 23:39:56.476549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.476407 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-metrics-certs\") pod \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " Nov 03 23:39:56.476549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.476486 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-kube-api-access-dvl6w\" (UniqueName: \"kubernetes.io/projected/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-kube-api-access-dvl6w\") pod \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " Nov 03 23:39:56.476549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.476529 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-maas-api-tls\") pod \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\" (UID: \"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c\") " Nov 03 23:39:56.476746 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.476710 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-openshift-service-ca" pod "6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" (UID: "6a320c23-b1f3-47f7-bbae-8198d1a0ce1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:39:56.476886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.476843 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.478586 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.478554 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-metrics-certs" pod "6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" (UID: "6a320c23-b1f3-47f7-bbae-8198d1a0ce1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:39:56.478586 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.478572 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-maas-api-tls" pod "6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" (UID: "6a320c23-b1f3-47f7-bbae-8198d1a0ce1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:39:56.478752 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.478604 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-kube-api-access-dvl6w" pod "6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" (UID: "6a320c23-b1f3-47f7-bbae-8198d1a0ce1c"). InnerVolumeSpecName "kube-api-access-dvl6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:56.498382 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.498352 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:56.577710 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.577678 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvl6w\" (UniqueName: \"kubernetes.io/projected/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-kube-api-access-dvl6w\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.577710 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.577707 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.577710 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.577717 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:56.652723 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.652688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:56.665524 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.665493 6208 generic.go:356] "Generic (PLEG): container finished" podID="6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" containerID="fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1" exitCode=0 Nov 03 23:39:56.665697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.665547 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" Nov 03 23:39:56.665697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.665573 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" event={"ID":"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c","Type":"ContainerDied","Data":"fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1"} Nov 03 23:39:56.665697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.665626 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" event={"ID":"6a320c23-b1f3-47f7-bbae-8198d1a0ce1c","Type":"ContainerDied","Data":"8da05ddd842c24dac25a4d506b8a2de0989c08ab8798bf11d6945eabfd7d96fc"} Nov 03 23:39:56.665697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.665656 6208 scope.go:122] "RemoveContainer" containerID="fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1" Nov 03 23:39:56.667495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.667473 6208 generic.go:356] "Generic (PLEG): container finished" podID="908dd01e-6450-44ea-987d-989cf05595a4" containerID="653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c" exitCode=0 Nov 03 23:39:56.667619 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.667505 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" event={"ID":"908dd01e-6450-44ea-987d-989cf05595a4","Type":"ContainerDied","Data":"653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c"} Nov 03 23:39:56.667619 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.667532 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" Nov 03 23:39:56.667619 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.667535 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv" event={"ID":"908dd01e-6450-44ea-987d-989cf05595a4","Type":"ContainerDied","Data":"9b042a265815ddf3a1fccbeee0541df046e6b6f9372a7a114402a9722cd09204"} Nov 03 23:39:56.689995 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.689979 6208 scope.go:122] "RemoveContainer" containerID="fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1" Nov 03 23:39:56.690324 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:56.690306 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1\": container with ID starting with fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1 not found: ID does not exist" containerID="fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1" Nov 03 23:39:56.690391 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.690340 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1"} err="failed to get container status \"fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1\": rpc error: code = NotFound desc = could not find container \"fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1\": container with ID starting with fc8e7f7d157bdcc339a19c32628478e45bd303839f591378fdd1af9352fe17f1 not found: ID does not exist" Nov 03 23:39:56.690391 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.690364 6208 scope.go:122] "RemoveContainer" containerID="653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c" Nov 03 23:39:56.699523 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.699501 6208 scope.go:122] "RemoveContainer" containerID="343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba" Nov 03 23:39:56.722325 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.722276 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv"] Nov 03 23:39:56.723940 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.723918 6208 scope.go:122] "RemoveContainer" containerID="653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c" Nov 03 23:39:56.728594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.728527 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-80942cbb-openshift-default-7979c64dd9-467bv"] Nov 03 23:39:56.731157 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:56.731126 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c\": container with ID starting with 653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c not found: ID does not exist" containerID="653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c" Nov 03 23:39:56.731313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.731163 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c"} err="failed to get container status \"653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c\": rpc error: code = NotFound desc = could not find container \"653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c\": container with ID starting with 653ac02b3a15dcba33b2915b8c74f29c34d2aa5a05bbb8578003d8e5e823d83c not found: ID does not exist" Nov 03 23:39:56.731313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.731189 6208 scope.go:122] "RemoveContainer" containerID="343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba" Nov 03 23:39:56.731545 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:56.731516 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba\": container with ID starting with 343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba not found: ID does not exist" containerID="343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba" Nov 03 23:39:56.731644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.731546 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba"} err="failed to get container status \"343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba\": rpc error: code = NotFound desc = could not find container \"343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba\": container with ID starting with 343126baed0b5cf10e923b6f72a96dde6c8a37c124b0a098a0c67570324c3bba not found: ID does not exist" Nov 03 23:39:56.745334 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.745310 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm"] Nov 03 23:39:56.748056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.748019 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm"] Nov 03 23:39:56.777569 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.777534 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:56.780787 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.780753 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" podUID="99b0b19e-75f6-47ba-b050-4c2055d5ec65" containerName="payload-processing" containerID="cri-o://1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3" gracePeriod=25 Nov 03 23:39:56.852499 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:56.852461 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" podUID="a29c3d84-f959-47f8-8bae-e69097b8ccdd" containerName="payload-pre-processing" containerID="cri-o://4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97" gracePeriod=25 Nov 03 23:39:57.087125 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.087021 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:57.152935 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.152900 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:57.174837 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.174812 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:57.185806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.185778 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:57.190629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.190603 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs"] Nov 03 23:39:57.191191 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191170 6208 cpu_manager.go:414] "containerMap: removing container" podUID="abbfb154-b4c1-4952-8423-fa03f425a559" containerName="payload-pre-processing" Nov 03 23:39:57.191269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191192 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbfb154-b4c1-4952-8423-fa03f425a559" containerName="payload-pre-processing" Nov 03 23:39:57.191269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191214 6208 cpu_manager.go:414] "containerMap: removing container" podUID="99b0b19e-75f6-47ba-b050-4c2055d5ec65" containerName="payload-processing" Nov 03 23:39:57.191269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191221 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b0b19e-75f6-47ba-b050-4c2055d5ec65" containerName="payload-processing" Nov 03 23:39:57.191269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191236 6208 cpu_manager.go:414] "containerMap: removing container" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" Nov 03 23:39:57.191269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191244 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" Nov 03 23:39:57.191269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191265 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5c8ea5c7-8174-402e-af0a-cda38241ce95" containerName="payload-processing" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191272 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8ea5c7-8174-402e-af0a-cda38241ce95" containerName="payload-processing" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191288 6208 cpu_manager.go:414] "containerMap: removing container" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191294 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191304 6208 cpu_manager.go:414] "containerMap: removing container" podUID="60dca8f6-9229-4ddc-bf78-227ddd3950b4" containerName="payload-processing" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191311 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60dca8f6-9229-4ddc-bf78-227ddd3950b4" containerName="payload-processing" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191325 6208 cpu_manager.go:414] "containerMap: removing container" podUID="6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" containerName="maas-api" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191332 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" containerName="maas-api" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191343 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f6f283b0-20cd-4bdc-8a33-285bf326d1bb" containerName="payload-pre-processing" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191350 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f283b0-20cd-4bdc-8a33-285bf326d1bb" containerName="payload-pre-processing" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191372 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a29c3d84-f959-47f8-8bae-e69097b8ccdd" containerName="payload-pre-processing" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191378 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29c3d84-f959-47f8-8bae-e69097b8ccdd" containerName="payload-pre-processing" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191389 6208 cpu_manager.go:414] "containerMap: removing container" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" Nov 03 23:39:57.191444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191397 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191496 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbfb154-b4c1-4952-8423-fa03f425a559" containerName="payload-pre-processing" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191511 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b0b19e-75f6-47ba-b050-4c2055d5ec65" containerName="payload-processing" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191520 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191529 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8ea5c7-8174-402e-af0a-cda38241ce95" containerName="payload-processing" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191540 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191548 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="60dca8f6-9229-4ddc-bf78-227ddd3950b4" containerName="payload-processing" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191558 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" containerName="maas-api" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191566 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f283b0-20cd-4bdc-8a33-285bf326d1bb" containerName="payload-pre-processing" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191573 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191583 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29c3d84-f959-47f8-8bae-e69097b8ccdd" containerName="payload-pre-processing" Nov 03 23:39:57.191817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.191591 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="908dd01e-6450-44ea-987d-989cf05595a4" containerName="istio-proxy" Nov 03 23:39:57.195078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.195055 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:39:57.205823 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.205799 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6"] Nov 03 23:39:57.206198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.206185 6208 cpu_manager.go:414] "containerMap: removing container" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" Nov 03 23:39:57.206251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.206199 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerName="istio-proxy" Nov 03 23:39:57.209205 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.209190 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:39:57.210869 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.210824 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-80f8ba-dockercfg-fk4nh\"" Nov 03 23:39:57.210869 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.210860 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-80f8ba\"" Nov 03 23:39:57.215444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.215420 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs"] Nov 03 23:39:57.226777 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.226751 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6"] Nov 03 23:39:57.260908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.260874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:57.285097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.284956 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-kube-api-access-zggdd\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-kube-api-access-zggdd\") pod \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " Nov 03 23:39:57.285097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285017 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-podinfo\") pod \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " Nov 03 23:39:57.285309 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285096 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-credential-socket\") pod \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " Nov 03 23:39:57.285309 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285130 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-data\") pod \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " Nov 03 23:39:57.285309 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285164 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istiod-ca-cert\") pod \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " Nov 03 23:39:57.285309 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285231 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-token\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-token\") pod \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " Nov 03 23:39:57.285309 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285266 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b0b19e-75f6-47ba-b050-4c2055d5ec65-kube-api-access-nhgbf\" (UniqueName: \"kubernetes.io/projected/99b0b19e-75f6-47ba-b050-4c2055d5ec65-kube-api-access-nhgbf\") pod \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\" (UID: \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\") " Nov 03 23:39:57.285562 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285350 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-socket\") pod \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " Nov 03 23:39:57.285562 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285387 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-envoy\") pod \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " Nov 03 23:39:57.285562 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285417 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29c3d84-f959-47f8-8bae-e69097b8ccdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29c3d84-f959-47f8-8bae-e69097b8ccdd-plugins-config-volume\") pod \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\" (UID: \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\") " Nov 03 23:39:57.285562 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285487 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-certs\") pod \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\" (UID: \"dfa30bbe-ebf1-4dba-b206-41223aa75c1d\") " Nov 03 23:39:57.285562 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285518 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99b0b19e-75f6-47ba-b050-4c2055d5ec65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b0b19e-75f6-47ba-b050-4c2055d5ec65-plugins-config-volume\") pod \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\" (UID: \"99b0b19e-75f6-47ba-b050-4c2055d5ec65\") " Nov 03 23:39:57.285800 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285579 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29c3d84-f959-47f8-8bae-e69097b8ccdd-kube-api-access-pxhb4\" (UniqueName: \"kubernetes.io/projected/a29c3d84-f959-47f8-8bae-e69097b8ccdd-kube-api-access-pxhb4\") pod \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\" (UID: \"a29c3d84-f959-47f8-8bae-e69097b8ccdd\") " Nov 03 23:39:57.285800 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285728 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e29d36-f359-48ed-bebb-15951255c782-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs\" (UID: \"b6e29d36-f359-48ed-bebb-15951255c782\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:39:57.285902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285860 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8srs\" (UniqueName: \"kubernetes.io/projected/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-kube-api-access-g8srs\") pod \"payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6\" (UID: \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:39:57.285953 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285930 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkj4\" (UniqueName: \"kubernetes.io/projected/b6e29d36-f359-48ed-bebb-15951255c782-kube-api-access-fxkj4\") pod \"payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs\" (UID: \"b6e29d36-f359-48ed-bebb-15951255c782\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:39:57.286010 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.285988 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6\" (UID: \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:39:57.288297 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.288130 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29c3d84-f959-47f8-8bae-e69097b8ccdd-plugins-config-volume" pod "a29c3d84-f959-47f8-8bae-e69097b8ccdd" (UID: "a29c3d84-f959-47f8-8bae-e69097b8ccdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:39:57.289267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.289209 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-kube-api-access-zggdd" pod "dfa30bbe-ebf1-4dba-b206-41223aa75c1d" (UID: "dfa30bbe-ebf1-4dba-b206-41223aa75c1d"). InnerVolumeSpecName "kube-api-access-zggdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:57.289267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.289232 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b0b19e-75f6-47ba-b050-4c2055d5ec65-plugins-config-volume" pod "99b0b19e-75f6-47ba-b050-4c2055d5ec65" (UID: "99b0b19e-75f6-47ba-b050-4c2055d5ec65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:39:57.289772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.289745 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-data" pod "dfa30bbe-ebf1-4dba-b206-41223aa75c1d" (UID: "dfa30bbe-ebf1-4dba-b206-41223aa75c1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:57.289982 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.289962 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-certs" pod "dfa30bbe-ebf1-4dba-b206-41223aa75c1d" (UID: "dfa30bbe-ebf1-4dba-b206-41223aa75c1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:57.290378 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.290354 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-token" pod "dfa30bbe-ebf1-4dba-b206-41223aa75c1d" (UID: "dfa30bbe-ebf1-4dba-b206-41223aa75c1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:57.291446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.291406 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29c3d84-f959-47f8-8bae-e69097b8ccdd-kube-api-access-pxhb4" pod "a29c3d84-f959-47f8-8bae-e69097b8ccdd" (UID: "a29c3d84-f959-47f8-8bae-e69097b8ccdd"). InnerVolumeSpecName "kube-api-access-pxhb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:57.292173 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.292149 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-podinfo" pod "dfa30bbe-ebf1-4dba-b206-41223aa75c1d" (UID: "dfa30bbe-ebf1-4dba-b206-41223aa75c1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:39:57.294092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.294064 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-socket" pod "dfa30bbe-ebf1-4dba-b206-41223aa75c1d" (UID: "dfa30bbe-ebf1-4dba-b206-41223aa75c1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:57.294281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.294261 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-credential-socket" pod "dfa30bbe-ebf1-4dba-b206-41223aa75c1d" (UID: "dfa30bbe-ebf1-4dba-b206-41223aa75c1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:57.295406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.295381 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istiod-ca-cert" pod "dfa30bbe-ebf1-4dba-b206-41223aa75c1d" (UID: "dfa30bbe-ebf1-4dba-b206-41223aa75c1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:39:57.295874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.295851 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b0b19e-75f6-47ba-b050-4c2055d5ec65-kube-api-access-nhgbf" pod "99b0b19e-75f6-47ba-b050-4c2055d5ec65" (UID: "99b0b19e-75f6-47ba-b050-4c2055d5ec65"). InnerVolumeSpecName "kube-api-access-nhgbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:39:57.297501 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.297478 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-envoy" pod "dfa30bbe-ebf1-4dba-b206-41223aa75c1d" (UID: "dfa30bbe-ebf1-4dba-b206-41223aa75c1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:39:57.300695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.300669 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" path="/var/lib/kubelet/pods/6a320c23-b1f3-47f7-bbae-8198d1a0ce1c/volumes" Nov 03 23:39:57.301222 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.301203 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908dd01e-6450-44ea-987d-989cf05595a4" path="/var/lib/kubelet/pods/908dd01e-6450-44ea-987d-989cf05595a4/volumes" Nov 03 23:39:57.387115 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387076 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkj4\" (UniqueName: \"kubernetes.io/projected/b6e29d36-f359-48ed-bebb-15951255c782-kube-api-access-fxkj4\") pod \"payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs\" (UID: \"b6e29d36-f359-48ed-bebb-15951255c782\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:39:57.387311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387142 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6\" (UID: \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:39:57.387311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387193 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e29d36-f359-48ed-bebb-15951255c782-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs\" (UID: \"b6e29d36-f359-48ed-bebb-15951255c782\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:39:57.387311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387283 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8srs\" (UniqueName: \"kubernetes.io/projected/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-kube-api-access-g8srs\") pod \"payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6\" (UID: \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387357 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387373 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b0b19e-75f6-47ba-b050-4c2055d5ec65-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387389 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxhb4\" (UniqueName: \"kubernetes.io/projected/a29c3d84-f959-47f8-8bae-e69097b8ccdd-kube-api-access-pxhb4\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387403 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zggdd\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-kube-api-access-zggdd\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387417 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387430 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387444 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387458 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387471 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387484 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhgbf\" (UniqueName: \"kubernetes.io/projected/99b0b19e-75f6-47ba-b050-4c2055d5ec65-kube-api-access-nhgbf\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387498 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.387507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387511 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa30bbe-ebf1-4dba-b206-41223aa75c1d-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.388235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.387523 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29c3d84-f959-47f8-8bae-e69097b8ccdd-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:39:57.388288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.388265 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6\" (UID: \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:39:57.388343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.388289 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e29d36-f359-48ed-bebb-15951255c782-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs\" (UID: \"b6e29d36-f359-48ed-bebb-15951255c782\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:39:57.409159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.409126 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8srs\" (UniqueName: \"kubernetes.io/projected/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-kube-api-access-g8srs\") pod \"payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6\" (UID: \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:39:57.409389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.409366 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkj4\" (UniqueName: \"kubernetes.io/projected/b6e29d36-f359-48ed-bebb-15951255c782-kube-api-access-fxkj4\") pod \"payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs\" (UID: \"b6e29d36-f359-48ed-bebb-15951255c782\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:39:57.473767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.473725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:57.511373 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.511343 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:39:57.531278 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.531239 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:39:57.677356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.677319 6208 generic.go:356] "Generic (PLEG): container finished" podID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" containerID="f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62" exitCode=0 Nov 03 23:39:57.677536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.677396 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" Nov 03 23:39:57.677536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.677397 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" event={"ID":"dfa30bbe-ebf1-4dba-b206-41223aa75c1d","Type":"ContainerDied","Data":"f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62"} Nov 03 23:39:57.677536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.677486 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt" event={"ID":"dfa30bbe-ebf1-4dba-b206-41223aa75c1d","Type":"ContainerDied","Data":"fc226a36dd19a3367148b0ac9da629ab40e59bbd8a1e4308da0bf59a4f23cc99"} Nov 03 23:39:57.677536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.677518 6208 scope.go:122] "RemoveContainer" containerID="f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62" Nov 03 23:39:57.678962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.678937 6208 generic.go:356] "Generic (PLEG): container finished" podID="99b0b19e-75f6-47ba-b050-4c2055d5ec65" containerID="1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3" exitCode=0 Nov 03 23:39:57.679071 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.679017 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" event={"ID":"99b0b19e-75f6-47ba-b050-4c2055d5ec65","Type":"ContainerDied","Data":"1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3"} Nov 03 23:39:57.679131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.679090 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" event={"ID":"99b0b19e-75f6-47ba-b050-4c2055d5ec65","Type":"ContainerDied","Data":"be5f8849f76b10b0c15c442eb4b4c47999bf48e4fcbc7806fdf97db6433dd45e"} Nov 03 23:39:57.679195 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.679181 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598" Nov 03 23:39:57.682473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.682452 6208 generic.go:356] "Generic (PLEG): container finished" podID="a29c3d84-f959-47f8-8bae-e69097b8ccdd" containerID="4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97" exitCode=0 Nov 03 23:39:57.682602 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.682487 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" event={"ID":"a29c3d84-f959-47f8-8bae-e69097b8ccdd","Type":"ContainerDied","Data":"4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97"} Nov 03 23:39:57.682602 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.682510 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" Nov 03 23:39:57.682602 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.682514 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd" event={"ID":"a29c3d84-f959-47f8-8bae-e69097b8ccdd","Type":"ContainerDied","Data":"be92381f61211e039f749d167fa9f88f4d2aefe8c936d375f7e626b8fad08eee"} Nov 03 23:39:57.709090 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.709066 6208 scope.go:122] "RemoveContainer" containerID="394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8" Nov 03 23:39:57.717981 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.717960 6208 scope.go:122] "RemoveContainer" containerID="f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62" Nov 03 23:39:57.718311 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:57.718289 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62\": container with ID starting with f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62 not found: ID does not exist" containerID="f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62" Nov 03 23:39:57.718359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.718321 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62"} err="failed to get container status \"f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62\": rpc error: code = NotFound desc = could not find container \"f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62\": container with ID starting with f45b7324d02701d5e50b14e9cf67fd610a88a40f67bd3488c0c743ffd36a4e62 not found: ID does not exist" Nov 03 23:39:57.718359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.718338 6208 scope.go:122] "RemoveContainer" containerID="394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8" Nov 03 23:39:57.718595 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:57.718581 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8\": container with ID starting with 394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8 not found: ID does not exist" containerID="394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8" Nov 03 23:39:57.718654 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.718596 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8"} err="failed to get container status \"394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8\": rpc error: code = NotFound desc = could not find container \"394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8\": container with ID starting with 394156682c9c20ce7508dd60cf345a398a3e3e341fffd8ed0c014b7672691cb8 not found: ID does not exist" Nov 03 23:39:57.718654 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.718607 6208 scope.go:122] "RemoveContainer" containerID="1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3" Nov 03 23:39:57.723654 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.723629 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt"] Nov 03 23:39:57.728351 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.728330 6208 scope.go:122] "RemoveContainer" containerID="1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3" Nov 03 23:39:57.728759 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:57.728733 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3\": container with ID starting with 1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3 not found: ID does not exist" containerID="1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3" Nov 03 23:39:57.728842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.728770 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3"} err="failed to get container status \"1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3\": rpc error: code = NotFound desc = could not find container \"1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3\": container with ID starting with 1a47270d712a4b3bfbd119ef7ea8545b9ba169a47ba593f2d99a3f12ca1fbce3 not found: ID does not exist" Nov 03 23:39:57.728842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.728792 6208 scope.go:122] "RemoveContainer" containerID="4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97" Nov 03 23:39:57.732085 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.732063 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7c2a2d28-openshift-default-554f5cd95-bj5gt"] Nov 03 23:39:57.737276 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.737255 6208 scope.go:122] "RemoveContainer" containerID="4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97" Nov 03 23:39:57.737553 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:39:57.737533 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97\": container with ID starting with 4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97 not found: ID does not exist" containerID="4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97" Nov 03 23:39:57.737607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.737561 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97"} err="failed to get container status \"4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97\": rpc error: code = NotFound desc = could not find container \"4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97\": container with ID starting with 4744befa7d44c35966fe8f348e92679e4256f050630bfd05da8f6de3a8fcca97 not found: ID does not exist" Nov 03 23:39:57.743437 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.743412 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598"] Nov 03 23:39:57.745174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.745155 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9e9be6c8-6cf47d54c6-dn598"] Nov 03 23:39:57.759476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.759447 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd"] Nov 03 23:39:57.762112 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.762089 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9e9be6c8-7b5596b495-2lsdd"] Nov 03 23:39:57.812190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.812164 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs"] Nov 03 23:39:57.813536 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:57.813511 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e29d36_f359_48ed_bebb_15951255c782.slice/crio-d08c7a5fb2316fe533464cb6345fcc299793f4027130fa2e99e21a0bf96fb36f WatchSource:0}: Error finding container d08c7a5fb2316fe533464cb6345fcc299793f4027130fa2e99e21a0bf96fb36f: Status 404 returned error can't find the container with id d08c7a5fb2316fe533464cb6345fcc299793f4027130fa2e99e21a0bf96fb36f Nov 03 23:39:57.843847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:57.843729 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6"] Nov 03 23:39:57.846146 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:57.846120 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3cc16a_8f73_49a2_8aec_87ebbb3cc1eb.slice/crio-8998e4382086b761cd7104be19113b1385b69d2bfeff41d4a61edb09a91e9771 WatchSource:0}: Error finding container 8998e4382086b761cd7104be19113b1385b69d2bfeff41d4a61edb09a91e9771: Status 404 returned error can't find the container with id 8998e4382086b761cd7104be19113b1385b69d2bfeff41d4a61edb09a91e9771 Nov 03 23:39:58.310299 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.310270 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:58.422765 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.422727 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:58.689423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.689333 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" event={"ID":"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb","Type":"ContainerStarted","Data":"7e6cd475fab8c6dd82cf71e37b1dc66468d8e5b7795bdca4deed4033685d49dc"} Nov 03 23:39:58.689423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.689390 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" event={"ID":"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb","Type":"ContainerStarted","Data":"8998e4382086b761cd7104be19113b1385b69d2bfeff41d4a61edb09a91e9771"} Nov 03 23:39:58.689667 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.689438 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:39:58.693664 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.693636 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" event={"ID":"b6e29d36-f359-48ed-bebb-15951255c782","Type":"ContainerStarted","Data":"ac3e94022d1240a491f96ab3d9b06fd91765b09e55f820bce2513f63235f316f"} Nov 03 23:39:58.693794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.693678 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" event={"ID":"b6e29d36-f359-48ed-bebb-15951255c782","Type":"ContainerStarted","Data":"d08c7a5fb2316fe533464cb6345fcc299793f4027130fa2e99e21a0bf96fb36f"} Nov 03 23:39:58.693794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.693729 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:39:58.731076 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.729185 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" podStartSLOduration=1.7291717599999998 podStartE2EDuration="1.72917176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:58.726948652 +0000 UTC m=+617.549544816" watchObservedRunningTime="2026-11-03 23:39:58.72917176 +0000 UTC m=+617.551767922" Nov 03 23:39:58.746620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.746591 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf"] Nov 03 23:39:58.750225 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.750208 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:58.764860 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.764815 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" podStartSLOduration=1.76480179 podStartE2EDuration="1.76480179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:58.762250306 +0000 UTC m=+617.584846478" watchObservedRunningTime="2026-11-03 23:39:58.76480179 +0000 UTC m=+617.587397952" Nov 03 23:39:58.780339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.780315 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f6f292-openshift-default-dockercfg-b884k\"" Nov 03 23:39:58.802160 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.802127 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf"] Nov 03 23:39:58.903728 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.903690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:58.904826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.904792 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:58.904826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.904831 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:58.905048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.904859 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:58.905048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.904883 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:58.905048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.904909 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6qr\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-kube-api-access-zq6qr\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:58.905048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.904927 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:58.905048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.904959 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:58.905048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.904986 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:58.905048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:58.905007 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e6f98-1a8e-4778-a9d9-074dce32314f-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.005782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.005744 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.005987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.005786 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.005987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.005816 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.005987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.005838 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.005987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.005867 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6qr\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-kube-api-access-zq6qr\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.005987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.005888 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.005987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.005923 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.006365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.006058 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.006365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.006121 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e6f98-1a8e-4778-a9d9-074dce32314f-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.006365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.006245 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.006519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.006399 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.006519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.006439 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.006626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.006528 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.006840 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.006818 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e6f98-1a8e-4778-a9d9-074dce32314f-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.008264 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.008233 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.008506 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.008489 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.019692 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.019663 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6qr\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-kube-api-access-zq6qr\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.024124 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.024101 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.063276 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.063241 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:39:59.087514 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.087487 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:39:59.292725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.292637 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e9be6c8-66547785c-wf7lm" podUID="6a320c23-b1f3-47f7-bbae-8198d1a0ce1c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:39:59.292901 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.292798 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b0b19e-75f6-47ba-b050-4c2055d5ec65" path="/var/lib/kubelet/pods/99b0b19e-75f6-47ba-b050-4c2055d5ec65/volumes" Nov 03 23:39:59.293230 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.293215 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29c3d84-f959-47f8-8bae-e69097b8ccdd" path="/var/lib/kubelet/pods/a29c3d84-f959-47f8-8bae-e69097b8ccdd/volumes" Nov 03 23:39:59.293625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.293612 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa30bbe-ebf1-4dba-b206-41223aa75c1d" path="/var/lib/kubelet/pods/dfa30bbe-ebf1-4dba-b206-41223aa75c1d/volumes" Nov 03 23:39:59.366207 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.366174 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf"] Nov 03 23:39:59.367700 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:39:59.367670 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769e6f98_1a8e_4778_a9d9_074dce32314f.slice/crio-74aa54aa323c782539e229073b5196c187216ec1cdea5635097ab2447427196b WatchSource:0}: Error finding container 74aa54aa323c782539e229073b5196c187216ec1cdea5635097ab2447427196b: Status 404 returned error can't find the container with id 74aa54aa323c782539e229073b5196c187216ec1cdea5635097ab2447427196b Nov 03 23:39:59.369991 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.369961 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:59.370091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.370075 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:59.370127 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.370113 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:39:59.702204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.701848 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" event={"ID":"769e6f98-1a8e-4778-a9d9-074dce32314f","Type":"ContainerStarted","Data":"0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84"} Nov 03 23:39:59.702204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.701959 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" event={"ID":"769e6f98-1a8e-4778-a9d9-074dce32314f","Type":"ContainerStarted","Data":"74aa54aa323c782539e229073b5196c187216ec1cdea5635097ab2447427196b"} Nov 03 23:39:59.803909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.803849 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" podStartSLOduration=1.803835815 podStartE2EDuration="1.803835815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:39:59.803283839 +0000 UTC m=+618.625880002" watchObservedRunningTime="2026-11-03 23:39:59.803835815 +0000 UTC m=+618.626431976" Nov 03 23:39:59.849812 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.849768 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg"] Nov 03 23:39:59.853406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.853388 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:39:59.863353 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.863330 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:39:59.863627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.863612 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:39:59.865000 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.864983 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-80f8ba-serving-cert\"" Nov 03 23:39:59.875949 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.875930 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:39:59.875949 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.875936 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xflt\"" Nov 03 23:39:59.920712 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:39:59.920679 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg"] Nov 03 23:40:00.016884 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.016849 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-metrics-certs\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.017024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.016895 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.017024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.016923 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn6fc\" (UniqueName: \"kubernetes.io/projected/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-kube-api-access-cn6fc\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.017144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.017083 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.054976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.054927 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:00.063914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.063887 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:40:00.069146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.069116 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:40:00.117718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.117687 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.117914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.117752 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-metrics-certs\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.117914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.117785 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.117914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.117807 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn6fc\" (UniqueName: \"kubernetes.io/projected/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-kube-api-access-cn6fc\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.118553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.118521 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.120455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.120432 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-metrics-certs\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.120535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.120471 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.136893 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.136860 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn6fc\" (UniqueName: \"kubernetes.io/projected/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-kube-api-access-cn6fc\") pod \"maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.165887 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.165852 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:00.234941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.234913 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:00.409981 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.409891 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:00.489621 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.489578 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg"] Nov 03 23:40:00.495055 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:00.495009 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0f1c2d_87d1_4478_9e3a_4eed792f66d5.slice/crio-bd3e56e3b3e48bdeb5d4cdb91ba7953848c11d85a8b0eb908f74618a26035762 WatchSource:0}: Error finding container bd3e56e3b3e48bdeb5d4cdb91ba7953848c11d85a8b0eb908f74618a26035762: Status 404 returned error can't find the container with id bd3e56e3b3e48bdeb5d4cdb91ba7953848c11d85a8b0eb908f74618a26035762 Nov 03 23:40:00.709086 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.709029 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" event={"ID":"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5","Type":"ContainerStarted","Data":"bd3e56e3b3e48bdeb5d4cdb91ba7953848c11d85a8b0eb908f74618a26035762"} Nov 03 23:40:00.709386 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.709369 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:40:00.710366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:00.710344 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:40:01.715808 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:01.715766 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" event={"ID":"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5","Type":"ContainerStarted","Data":"844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba"} Nov 03 23:40:01.716247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:01.715982 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:01.806344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:01.806291 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" podStartSLOduration=2.369806762 podStartE2EDuration="2.806276313s" totalImagesPullingTime="436.469551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:01.794280693 +0000 UTC m=+620.616876855" watchObservedRunningTime="2026-11-03 23:40:01.806276313 +0000 UTC m=+620.628872475" Nov 03 23:40:01.879640 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:01.879599 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:02.630791 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:02.630757 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:02.985499 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:02.985462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:03.317178 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:03.317080 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:03.900343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:03.900301 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:04.199616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.199528 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:04.329950 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.329913 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9"] Nov 03 23:40:04.333628 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.333607 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:04.370053 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.369608 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62d2e16-32d3-46b2-8795-4fed31a1c458-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9\" (UID: \"b62d2e16-32d3-46b2-8795-4fed31a1c458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:04.370053 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.369697 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjqtb\" (UniqueName: \"kubernetes.io/projected/b62d2e16-32d3-46b2-8795-4fed31a1c458-kube-api-access-qjqtb\") pod \"payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9\" (UID: \"b62d2e16-32d3-46b2-8795-4fed31a1c458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:04.370053 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.369858 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f6f292\"" Nov 03 23:40:04.376630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.376600 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9"] Nov 03 23:40:04.377024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.376939 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f6f292-dockercfg-hphlt\"" Nov 03 23:40:04.470868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.470829 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjqtb\" (UniqueName: \"kubernetes.io/projected/b62d2e16-32d3-46b2-8795-4fed31a1c458-kube-api-access-qjqtb\") pod \"payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9\" (UID: \"b62d2e16-32d3-46b2-8795-4fed31a1c458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:04.471073 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.470982 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62d2e16-32d3-46b2-8795-4fed31a1c458-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9\" (UID: \"b62d2e16-32d3-46b2-8795-4fed31a1c458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:04.471700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.471681 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62d2e16-32d3-46b2-8795-4fed31a1c458-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9\" (UID: \"b62d2e16-32d3-46b2-8795-4fed31a1c458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:04.510573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.510530 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t"] Nov 03 23:40:04.523477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.523443 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjqtb\" (UniqueName: \"kubernetes.io/projected/b62d2e16-32d3-46b2-8795-4fed31a1c458-kube-api-access-qjqtb\") pod \"payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9\" (UID: \"b62d2e16-32d3-46b2-8795-4fed31a1c458\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:04.531242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.531197 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:04.566404 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.566362 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t"] Nov 03 23:40:04.571943 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.571907 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127f6328-65eb-4bea-898b-f01e05d6a6dc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t\" (UID: \"127f6328-65eb-4bea-898b-f01e05d6a6dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:04.571943 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.571944 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfmg\" (UniqueName: \"kubernetes.io/projected/127f6328-65eb-4bea-898b-f01e05d6a6dc-kube-api-access-mmfmg\") pod \"payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t\" (UID: \"127f6328-65eb-4bea-898b-f01e05d6a6dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:04.590389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.590332 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:04.648478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.648445 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:04.672633 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.672603 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127f6328-65eb-4bea-898b-f01e05d6a6dc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t\" (UID: \"127f6328-65eb-4bea-898b-f01e05d6a6dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:04.672758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.672640 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfmg\" (UniqueName: \"kubernetes.io/projected/127f6328-65eb-4bea-898b-f01e05d6a6dc-kube-api-access-mmfmg\") pod \"payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t\" (UID: \"127f6328-65eb-4bea-898b-f01e05d6a6dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:04.673583 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.673558 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127f6328-65eb-4bea-898b-f01e05d6a6dc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t\" (UID: \"127f6328-65eb-4bea-898b-f01e05d6a6dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:04.804518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.804433 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfmg\" (UniqueName: \"kubernetes.io/projected/127f6328-65eb-4bea-898b-f01e05d6a6dc-kube-api-access-mmfmg\") pod \"payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t\" (UID: \"127f6328-65eb-4bea-898b-f01e05d6a6dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:04.846007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:04.845971 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:05.056620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.056537 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9"] Nov 03 23:40:05.063913 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:05.063886 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62d2e16_32d3_46b2_8795_4fed31a1c458.slice/crio-2ce9a8e0f60e3350b0f3a6b03935a645a9d42f092475272dcb80b4fd213d41dd WatchSource:0}: Error finding container 2ce9a8e0f60e3350b0f3a6b03935a645a9d42f092475272dcb80b4fd213d41dd: Status 404 returned error can't find the container with id 2ce9a8e0f60e3350b0f3a6b03935a645a9d42f092475272dcb80b4fd213d41dd Nov 03 23:40:05.208066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.208023 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t"] Nov 03 23:40:05.208849 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:05.208828 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127f6328_65eb_4bea_898b_f01e05d6a6dc.slice/crio-239ee047da9b42ae5a6be3abbf403f23c9e8d74b1b82db637f4395674a129990 WatchSource:0}: Error finding container 239ee047da9b42ae5a6be3abbf403f23c9e8d74b1b82db637f4395674a129990: Status 404 returned error can't find the container with id 239ee047da9b42ae5a6be3abbf403f23c9e8d74b1b82db637f4395674a129990 Nov 03 23:40:05.349855 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.349749 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:05.354016 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.353989 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws"] Nov 03 23:40:05.354237 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.354217 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" containerID="cri-o://981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f" gracePeriod=30 Nov 03 23:40:05.558697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.558662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:05.704146 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:40:05.704025 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:40:05.741395 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.741346 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" event={"ID":"b62d2e16-32d3-46b2-8795-4fed31a1c458","Type":"ContainerStarted","Data":"4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1"} Nov 03 23:40:05.741604 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.741422 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" event={"ID":"b62d2e16-32d3-46b2-8795-4fed31a1c458","Type":"ContainerStarted","Data":"2ce9a8e0f60e3350b0f3a6b03935a645a9d42f092475272dcb80b4fd213d41dd"} Nov 03 23:40:05.741604 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.741550 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:05.742859 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.742839 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" event={"ID":"127f6328-65eb-4bea-898b-f01e05d6a6dc","Type":"ContainerStarted","Data":"8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6"} Nov 03 23:40:05.742915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.742882 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" event={"ID":"127f6328-65eb-4bea-898b-f01e05d6a6dc","Type":"ContainerStarted","Data":"239ee047da9b42ae5a6be3abbf403f23c9e8d74b1b82db637f4395674a129990"} Nov 03 23:40:05.742915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.742903 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:05.825398 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.825347 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" podStartSLOduration=1.825333976 podStartE2EDuration="1.825333976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:05.823715926 +0000 UTC m=+624.646312089" watchObservedRunningTime="2026-11-03 23:40:05.825333976 +0000 UTC m=+624.647930138" Nov 03 23:40:05.888166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:05.888116 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" podStartSLOduration=1.888101508 podStartE2EDuration="1.888101508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:05.887627002 +0000 UTC m=+624.710223164" watchObservedRunningTime="2026-11-03 23:40:05.888101508 +0000 UTC m=+624.710697669" Nov 03 23:40:06.038669 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:06.038628 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt"] Nov 03 23:40:06.183295 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:06.183260 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:06.621544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:06.621507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:07.467745 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:07.467714 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:07.727217 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:07.727145 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 03 23:40:08.245599 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:08.245564 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:08.508776 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:08.508688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:09.659616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.659576 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:09.705734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.705703 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 03 23:40:09.705902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.705774 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 03 23:40:09.844564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.844517 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv"] Nov 03 23:40:09.849342 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.849314 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:09.890785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.890756 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f6f292-serving-cert\"" Nov 03 23:40:09.912629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.912540 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204d311a-3628-48a3-98f1-36cf66bc12bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:09.912629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.912593 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndq5\" (UniqueName: \"kubernetes.io/projected/204d311a-3628-48a3-98f1-36cf66bc12bc-kube-api-access-2ndq5\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:09.912847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.912703 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-metrics-certs\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:09.912847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.912784 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:09.957498 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:09.957462 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv"] Nov 03 23:40:10.013698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.013666 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:10.013894 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.013765 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204d311a-3628-48a3-98f1-36cf66bc12bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:10.013894 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.013821 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndq5\" (UniqueName: \"kubernetes.io/projected/204d311a-3628-48a3-98f1-36cf66bc12bc-kube-api-access-2ndq5\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:10.013894 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.013853 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-metrics-certs\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:10.014673 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.014643 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204d311a-3628-48a3-98f1-36cf66bc12bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:10.016234 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.016205 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-metrics-certs\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:10.016321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.016233 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:10.029627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.029600 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndq5\" (UniqueName: \"kubernetes.io/projected/204d311a-3628-48a3-98f1-36cf66bc12bc-kube-api-access-2ndq5\") pod \"maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:10.164373 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.164291 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:10.182403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.182372 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:10.483525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.481513 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv"] Nov 03 23:40:10.486228 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:10.486203 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204d311a_3628_48a3_98f1_36cf66bc12bc.slice/crio-358fd090045037914cdd2fd29ce3e834ac49802f4701163878c5fa6d47cedbd2 WatchSource:0}: Error finding container 358fd090045037914cdd2fd29ce3e834ac49802f4701163878c5fa6d47cedbd2: Status 404 returned error can't find the container with id 358fd090045037914cdd2fd29ce3e834ac49802f4701163878c5fa6d47cedbd2 Nov 03 23:40:10.597001 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.596978 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:40:10.622883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.622856 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-credential-socket\") pod \"4ddc4913-62bd-41d6-be48-858dc92d8918\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " Nov 03 23:40:10.623064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.622895 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-socket\") pod \"4ddc4913-62bd-41d6-be48-858dc92d8918\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " Nov 03 23:40:10.623064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.622928 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-envoy\") pod \"4ddc4913-62bd-41d6-be48-858dc92d8918\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " Nov 03 23:40:10.623064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623013 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-kube-api-access-h8zvr\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-kube-api-access-h8zvr\") pod \"4ddc4913-62bd-41d6-be48-858dc92d8918\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " Nov 03 23:40:10.623206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623069 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-token\") pod \"4ddc4913-62bd-41d6-be48-858dc92d8918\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " Nov 03 23:40:10.623206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623102 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ddc4913-62bd-41d6-be48-858dc92d8918-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc4913-62bd-41d6-be48-858dc92d8918-istiod-ca-cert\") pod \"4ddc4913-62bd-41d6-be48-858dc92d8918\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " Nov 03 23:40:10.623206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623135 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-certs\") pod \"4ddc4913-62bd-41d6-be48-858dc92d8918\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " Nov 03 23:40:10.623206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623190 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-podinfo\") pod \"4ddc4913-62bd-41d6-be48-858dc92d8918\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " Nov 03 23:40:10.623206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623194 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-credential-socket" pod "4ddc4913-62bd-41d6-be48-858dc92d8918" (UID: "4ddc4913-62bd-41d6-be48-858dc92d8918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:40:10.623527 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623219 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-socket" pod "4ddc4913-62bd-41d6-be48-858dc92d8918" (UID: "4ddc4913-62bd-41d6-be48-858dc92d8918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:40:10.623527 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623227 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-data\") pod \"4ddc4913-62bd-41d6-be48-858dc92d8918\" (UID: \"4ddc4913-62bd-41d6-be48-858dc92d8918\") " Nov 03 23:40:10.623527 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623427 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-data" pod "4ddc4913-62bd-41d6-be48-858dc92d8918" (UID: "4ddc4913-62bd-41d6-be48-858dc92d8918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:40:10.623527 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623501 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-certs" pod "4ddc4913-62bd-41d6-be48-858dc92d8918" (UID: "4ddc4913-62bd-41d6-be48-858dc92d8918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:40:10.623694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623616 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddc4913-62bd-41d6-be48-858dc92d8918-istiod-ca-cert" pod "4ddc4913-62bd-41d6-be48-858dc92d8918" (UID: "4ddc4913-62bd-41d6-be48-858dc92d8918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:40:10.623736 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623698 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:10.623736 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623712 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:10.623736 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623726 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddc4913-62bd-41d6-be48-858dc92d8918-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:10.623858 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623737 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:10.623858 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.623746 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:10.625895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.625872 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-envoy" pod "4ddc4913-62bd-41d6-be48-858dc92d8918" (UID: "4ddc4913-62bd-41d6-be48-858dc92d8918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:40:10.625993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.625905 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-kube-api-access-h8zvr" pod "4ddc4913-62bd-41d6-be48-858dc92d8918" (UID: "4ddc4913-62bd-41d6-be48-858dc92d8918"). InnerVolumeSpecName "kube-api-access-h8zvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:40:10.625993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.625915 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-podinfo" pod "4ddc4913-62bd-41d6-be48-858dc92d8918" (UID: "4ddc4913-62bd-41d6-be48-858dc92d8918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:40:10.625993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.625928 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-token" pod "4ddc4913-62bd-41d6-be48-858dc92d8918" (UID: "4ddc4913-62bd-41d6-be48-858dc92d8918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:40:10.725251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.725170 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8zvr\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-kube-api-access-h8zvr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:10.725251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.725198 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:10.725251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.725207 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:10.725251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.725218 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddc4913-62bd-41d6-be48-858dc92d8918-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:10.762522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.762488 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:10.772872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.772844 6208 generic.go:356] "Generic (PLEG): container finished" podID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerID="981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f" exitCode=0 Nov 03 23:40:10.773010 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.772896 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" Nov 03 23:40:10.773010 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.772938 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" event={"ID":"4ddc4913-62bd-41d6-be48-858dc92d8918","Type":"ContainerDied","Data":"981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f"} Nov 03 23:40:10.773010 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.773001 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws" event={"ID":"4ddc4913-62bd-41d6-be48-858dc92d8918","Type":"ContainerDied","Data":"52926d1002e98a11a9f8225ee857139981145c5dd684bd4686b1b4f6aade99c9"} Nov 03 23:40:10.773180 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.773061 6208 scope.go:122] "RemoveContainer" containerID="981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f" Nov 03 23:40:10.774359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.774335 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" event={"ID":"204d311a-3628-48a3-98f1-36cf66bc12bc","Type":"ContainerStarted","Data":"358fd090045037914cdd2fd29ce3e834ac49802f4701163878c5fa6d47cedbd2"} Nov 03 23:40:10.783374 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.783356 6208 scope.go:122] "RemoveContainer" containerID="6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339" Nov 03 23:40:10.794578 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.794558 6208 scope.go:122] "RemoveContainer" containerID="981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f" Nov 03 23:40:10.794874 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:40:10.794857 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f\": container with ID starting with 981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f not found: ID does not exist" containerID="981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f" Nov 03 23:40:10.794963 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.794883 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f"} err="failed to get container status \"981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f\": rpc error: code = NotFound desc = could not find container \"981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f\": container with ID starting with 981941c5c956b10e344a96db9966598ba71242fcd5510197f624763123accb7f not found: ID does not exist" Nov 03 23:40:10.794963 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.794899 6208 scope.go:122] "RemoveContainer" containerID="6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339" Nov 03 23:40:10.795206 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:40:10.795177 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339\": container with ID starting with 6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339 not found: ID does not exist" containerID="6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339" Nov 03 23:40:10.795263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.795217 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339"} err="failed to get container status \"6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339\": rpc error: code = NotFound desc = could not find container \"6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339\": container with ID starting with 6c6b6fee04898766f12355c3658e790093665f84fdb843661e8cb9cb5655c339 not found: ID does not exist" Nov 03 23:40:10.825838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.825814 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws"] Nov 03 23:40:10.845878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:10.845857 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-044f4dcd-openshift-default-6664f7f849-w9zws"] Nov 03 23:40:11.039288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.039237 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" podUID="feb5ccde-6895-44ef-a0cc-d40d2ef42ae1" containerName="payload-processing" containerID="cri-o://3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04" gracePeriod=25 Nov 03 23:40:11.294262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.294234 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" path="/var/lib/kubelet/pods/4ddc4913-62bd-41d6-be48-858dc92d8918/volumes" Nov 03 23:40:11.300751 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.300728 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:40:11.330354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.330329 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-kube-api-access-h929s\") pod \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\" (UID: \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\") " Nov 03 23:40:11.330471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.330392 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-plugins-config-volume\") pod \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\" (UID: \"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1\") " Nov 03 23:40:11.330731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.330711 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-plugins-config-volume" pod "feb5ccde-6895-44ef-a0cc-d40d2ef42ae1" (UID: "feb5ccde-6895-44ef-a0cc-d40d2ef42ae1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:40:11.332400 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.332376 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-kube-api-access-h929s" pod "feb5ccde-6895-44ef-a0cc-d40d2ef42ae1" (UID: "feb5ccde-6895-44ef-a0cc-d40d2ef42ae1"). InnerVolumeSpecName "kube-api-access-h929s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:40:11.431387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.431354 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:11.431387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.431381 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h929s\" (UniqueName: \"kubernetes.io/projected/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1-kube-api-access-h929s\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:40:11.780960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.780923 6208 generic.go:356] "Generic (PLEG): container finished" podID="feb5ccde-6895-44ef-a0cc-d40d2ef42ae1" containerID="3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04" exitCode=0 Nov 03 23:40:11.781483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.780969 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" Nov 03 23:40:11.781483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.781006 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" event={"ID":"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1","Type":"ContainerDied","Data":"3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04"} Nov 03 23:40:11.781483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.781076 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt" event={"ID":"feb5ccde-6895-44ef-a0cc-d40d2ef42ae1","Type":"ContainerDied","Data":"7a81af34ff4e2633821dc0313d320e2e2e66f94bc90eef82221b756ad6809fc2"} Nov 03 23:40:11.781483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.781099 6208 scope.go:122] "RemoveContainer" containerID="3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04" Nov 03 23:40:11.783940 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.783918 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" event={"ID":"204d311a-3628-48a3-98f1-36cf66bc12bc","Type":"ContainerStarted","Data":"5939f7df0e40d1770a11e9c8e0b5dde2a19e7cef0434a2f802a894597304bc07"} Nov 03 23:40:11.784081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.784058 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:11.791696 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.791679 6208 scope.go:122] "RemoveContainer" containerID="3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04" Nov 03 23:40:11.791960 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:40:11.791942 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04\": container with ID starting with 3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04 not found: ID does not exist" containerID="3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04" Nov 03 23:40:11.792007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.791967 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04"} err="failed to get container status \"3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04\": rpc error: code = NotFound desc = could not find container \"3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04\": container with ID starting with 3542d81c4eeb38a772af082f902bb15b97d6749d43b6bd856ed2091d5e964c04 not found: ID does not exist" Nov 03 23:40:11.828697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.828667 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt"] Nov 03 23:40:11.836416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.836397 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d3157179-7484c755b6-7m9tt"] Nov 03 23:40:11.876398 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:11.876352 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" podStartSLOduration=2.456565397 podStartE2EDuration="2.876340733s" totalImagesPullingTime="419.775336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:11.867765415 +0000 UTC m=+630.690361578" watchObservedRunningTime="2026-11-03 23:40:11.876340733 +0000 UTC m=+630.698936895" Nov 03 23:40:12.308352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:12.308311 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:13.133426 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:13.133387 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:13.296662 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:13.296626 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb5ccde-6895-44ef-a0cc-d40d2ef42ae1" path="/var/lib/kubelet/pods/feb5ccde-6895-44ef-a0cc-d40d2ef42ae1/volumes" Nov 03 23:40:13.522077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:13.522014 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:14.516121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.516077 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:14.944316 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.944233 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:14.989693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.989663 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb"] Nov 03 23:40:14.990141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.990126 6208 cpu_manager.go:414] "containerMap: removing container" podUID="feb5ccde-6895-44ef-a0cc-d40d2ef42ae1" containerName="payload-processing" Nov 03 23:40:14.990188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.990143 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb5ccde-6895-44ef-a0cc-d40d2ef42ae1" containerName="payload-processing" Nov 03 23:40:14.990188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.990159 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" Nov 03 23:40:14.990188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.990164 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" Nov 03 23:40:14.990188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.990184 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" Nov 03 23:40:14.990188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.990190 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" Nov 03 23:40:14.990360 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.990255 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb5ccde-6895-44ef-a0cc-d40d2ef42ae1" containerName="payload-processing" Nov 03 23:40:14.990360 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.990266 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" Nov 03 23:40:14.990360 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.990275 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddc4913-62bd-41d6-be48-858dc92d8918" containerName="istio-proxy" Nov 03 23:40:14.994565 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:14.994549 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.032519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.032494 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c60893-openshift-default-dockercfg-hwjgp\"" Nov 03 23:40:15.032921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.032900 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb"] Nov 03 23:40:15.066250 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.066217 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.066393 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.066254 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.066393 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.066288 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.066514 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.066471 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7b2g\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-kube-api-access-h7b2g\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.066578 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.066512 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.066632 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.066573 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.066632 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.066614 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.066736 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.066703 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.066795 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.066736 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.167909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.167872 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.167909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.167909 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.167952 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.167974 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168022 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168077 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168110 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168441 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168184 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7b2g\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-kube-api-access-h7b2g\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168441 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168205 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168441 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168387 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168598 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168469 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168598 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168480 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.168906 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168876 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.169007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.168964 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.170597 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.170572 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.170981 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.170959 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.185328 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.185299 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.185328 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.185317 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7b2g\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-kube-api-access-h7b2g\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.220238 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.220202 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:15.307365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.307333 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:15.594238 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.594213 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb"] Nov 03 23:40:15.596394 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:15.596368 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbdb3b7_b3c7_4908_8ced_a4185bafa586.slice/crio-1f29a2b776ca276e55a8abe5d7b0a3a87b9f1e75247a665532f5c481f30c7c79 WatchSource:0}: Error finding container 1f29a2b776ca276e55a8abe5d7b0a3a87b9f1e75247a665532f5c481f30c7c79: Status 404 returned error can't find the container with id 1f29a2b776ca276e55a8abe5d7b0a3a87b9f1e75247a665532f5c481f30c7c79 Nov 03 23:40:15.600476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.599982 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:15.600476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.600103 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:15.600476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.600149 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:15.808659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.808614 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" event={"ID":"adbdb3b7-b3c7-4908-8ced-a4185bafa586","Type":"ContainerStarted","Data":"71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020"} Nov 03 23:40:15.808856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.808701 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" event={"ID":"adbdb3b7-b3c7-4908-8ced-a4185bafa586","Type":"ContainerStarted","Data":"1f29a2b776ca276e55a8abe5d7b0a3a87b9f1e75247a665532f5c481f30c7c79"} Nov 03 23:40:15.851808 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:15.851713 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" podStartSLOduration=1.851700084 podStartE2EDuration="1.851700084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:15.850178321 +0000 UTC m=+634.672774486" watchObservedRunningTime="2026-11-03 23:40:15.851700084 +0000 UTC m=+634.674296247" Nov 03 23:40:16.307107 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:16.307070 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:16.307820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:16.307801 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:16.313531 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:16.313508 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:16.539379 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:16.539343 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:16.751626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:16.751591 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 03 23:40:16.752129 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:16.751788 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 03 23:40:16.813805 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:16.813769 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:16.814872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:16.814853 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:16.868056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:16.868000 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:17.041882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:17.041800 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:17.798718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:17.798694 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 03 23:40:18.214653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:18.214621 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:18.408343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:18.408308 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:18.586592 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:18.586478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:19.687338 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:19.687255 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:20.281898 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.281856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:20.376945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.376899 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld"] Nov 03 23:40:20.388363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.388336 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:20.388856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.388770 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld"] Nov 03 23:40:20.402012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.401984 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:20.425359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.425330 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6rt\" (UniqueName: \"kubernetes.io/projected/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-kube-api-access-6d6rt\") pod \"payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld\" (UID: \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:20.425528 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.425370 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld\" (UID: \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:20.438349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.438321 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-c60893\"" Nov 03 23:40:20.438700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.438669 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-c60893-dockercfg-r2jwz\"" Nov 03 23:40:20.526319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.526286 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6rt\" (UniqueName: \"kubernetes.io/projected/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-kube-api-access-6d6rt\") pod \"payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld\" (UID: \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:20.526319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.526323 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld\" (UID: \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:20.527062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.527010 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld\" (UID: \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:20.557099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.553459 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6rt\" (UniqueName: \"kubernetes.io/projected/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-kube-api-access-6d6rt\") pod \"payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld\" (UID: \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:20.704798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.704767 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:20.940165 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:20.940066 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:21.019990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:21.019961 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld"] Nov 03 23:40:21.020983 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:21.020952 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d8e725_2cb7_40a0_9acc_bb36b01a0fbd.slice/crio-e802afa080ca463517ea7aff398377556f451c56d92891efa2b434fedac06418 WatchSource:0}: Error finding container e802afa080ca463517ea7aff398377556f451c56d92891efa2b434fedac06418: Status 404 returned error can't find the container with id e802afa080ca463517ea7aff398377556f451c56d92891efa2b434fedac06418 Nov 03 23:40:21.246183 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:21.246142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:21.392210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:21.392175 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:21.846441 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:21.846402 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" event={"ID":"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd","Type":"ContainerStarted","Data":"722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba"} Nov 03 23:40:21.846863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:21.846473 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" event={"ID":"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd","Type":"ContainerStarted","Data":"e802afa080ca463517ea7aff398377556f451c56d92891efa2b434fedac06418"} Nov 03 23:40:21.846863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:21.846538 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:21.877844 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:21.877765 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" podStartSLOduration=1.877746447 podStartE2EDuration="1.877746447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:21.875874325 +0000 UTC m=+640.698470487" watchObservedRunningTime="2026-11-03 23:40:21.877746447 +0000 UTC m=+640.700342607" Nov 03 23:40:22.331618 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:22.331580 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:22.542330 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:22.542295 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:22.720491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:22.720459 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:23.244563 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:23.244523 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:23.379239 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:23.379204 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:23.747949 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:23.747918 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:23.937561 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:23.937524 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:24.761131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:24.761097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:25.767532 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:25.767502 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:26.893241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:26.893201 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:27.786286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:27.786249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:27.893385 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:27.893356 6208 generic.go:356] "Generic (PLEG): container finished" podID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerID="0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84" exitCode=0 Nov 03 23:40:27.893763 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:27.893432 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" event={"ID":"769e6f98-1a8e-4778-a9d9-074dce32314f","Type":"ContainerDied","Data":"0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84"} Nov 03 23:40:27.893839 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:27.893821 6208 scope.go:122] "RemoveContainer" containerID="0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84" Nov 03 23:40:27.895247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:27.895215 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:27.895365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:27.895299 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:27.895365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:27.895349 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:28.194707 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.194619 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp"] Nov 03 23:40:28.199105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.199079 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.205335 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.205234 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"default-dockercfg-926zp\"" Nov 03 23:40:28.205522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.205339 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"simulator-w0-f6f292-kserve-self-signed-certs\"" Nov 03 23:40:28.209214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.208695 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"kube-root-ca.crt\"" Nov 03 23:40:28.211856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.211833 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f6f292\"/\"openshift-service-ca.crt\"" Nov 03 23:40:28.220073 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.220046 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp"] Nov 03 23:40:28.284153 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.284112 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:28.344725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.344694 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ltt\" (UniqueName: \"kubernetes.io/projected/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-kube-api-access-m4ltt\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.344909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.344753 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tmp-dir\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.344967 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.344907 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-dshm\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.345013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.344992 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tls-certs\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.345112 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.345086 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-model-cache\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.345174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.345162 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-home\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.446284 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.446200 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tmp-dir\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.446284 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.446260 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-dshm\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.446549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.446307 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tls-certs\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.446549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.446347 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-model-cache\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.446549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.446415 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-home\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.446549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.446470 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ltt\" (UniqueName: \"kubernetes.io/projected/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-kube-api-access-m4ltt\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.446774 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.446655 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tmp-dir\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.446827 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.446766 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-model-cache\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.446827 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.446803 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-home\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.448653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.448629 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-dshm\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.448884 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.448865 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tls-certs\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.488094 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.488004 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ltt\" (UniqueName: \"kubernetes.io/projected/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-kube-api-access-m4ltt\") pod \"simulator-w0-f6f292-kserve-74858bcf8-rjkcp\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.515383 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.515347 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:28.876059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.875948 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp"] Nov 03 23:40:28.907089 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:28.903218 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88cecb6a_ebee_4eb2_abac_1cc23957ae3d.slice/crio-0ae7114ea2445e1bb6c5f974646b47106ecd8c124335ae9ff5ba36c0181fe715 WatchSource:0}: Error finding container 0ae7114ea2445e1bb6c5f974646b47106ecd8c124335ae9ff5ba36c0181fe715: Status 404 returned error can't find the container with id 0ae7114ea2445e1bb6c5f974646b47106ecd8c124335ae9ff5ba36c0181fe715 Nov 03 23:40:28.915667 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:28.915640 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" event={"ID":"769e6f98-1a8e-4778-a9d9-074dce32314f","Type":"ContainerStarted","Data":"5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5"} Nov 03 23:40:29.021761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:29.021731 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:29.074057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:29.073321 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:40:29.096059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:29.094169 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 23:40:29.096059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:29.094236 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 23:40:29.911660 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:29.911616 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:29.922815 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:29.922780 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" event={"ID":"88cecb6a-ebee-4eb2-abac-1cc23957ae3d","Type":"ContainerStarted","Data":"7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3"} Nov 03 23:40:29.922984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:29.922826 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" event={"ID":"88cecb6a-ebee-4eb2-abac-1cc23957ae3d","Type":"ContainerStarted","Data":"0ae7114ea2445e1bb6c5f974646b47106ecd8c124335ae9ff5ba36c0181fe715"} Nov 03 23:40:29.945489 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:29.945444 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podStartSLOduration=1.9454298589999999 podStartE2EDuration="1.945429859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:29.942994535 +0000 UTC m=+648.765590691" watchObservedRunningTime="2026-11-03 23:40:29.945429859 +0000 UTC m=+648.768026041" Nov 03 23:40:30.074059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:30.071469 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 23:40:30.074059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:30.071541 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 23:40:30.516495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:30.516459 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:30.709510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:30.709469 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:40:30.822680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:30.822586 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:30.952618 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:30.952573 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:31.069058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:31.065494 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 23:40:31.069058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:31.065553 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 23:40:32.116522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:32.116478 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 23:40:32.117082 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:32.116544 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 23:40:32.142747 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:32.142720 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:32.295058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:32.290601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:32.424832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:32.424654 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:32.896337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:32.896302 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 03 23:40:33.078059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:33.071420 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 23:40:33.078059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:33.071480 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 23:40:33.118089 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:33.118058 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:33.585282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:33.585251 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:33.779981 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:33.779940 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:33.909388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:33.909287 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:34.628282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:34.628241 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:34.734464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:34.734432 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:40:34.735657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:34.735632 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 03 23:40:35.569815 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.569765 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf"] Nov 03 23:40:35.574493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.574461 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.585457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.585424 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf"] Nov 03 23:40:35.630370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.630338 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:35.655578 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.655549 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5cb349-serving-cert\"" Nov 03 23:40:35.670873 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.670841 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1824481f-5659-4932-ab0b-6726f418b3a0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.671013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.670898 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.671109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.671017 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt8bm\" (UniqueName: \"kubernetes.io/projected/1824481f-5659-4932-ab0b-6726f418b3a0-kube-api-access-pt8bm\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.671170 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.671150 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-metrics-certs\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.704853 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:40:35.704816 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:40:35.744123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.744078 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:35.772752 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.772714 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-metrics-certs\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.772986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.772773 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1824481f-5659-4932-ab0b-6726f418b3a0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.772986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.772899 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.774196 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.774160 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1824481f-5659-4932-ab0b-6726f418b3a0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.775389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.775367 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt8bm\" (UniqueName: \"kubernetes.io/projected/1824481f-5659-4932-ab0b-6726f418b3a0-kube-api-access-pt8bm\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.777132 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.777110 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9"] Nov 03 23:40:35.782049 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.779786 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.782049 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.780143 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-metrics-certs\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.794957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.794930 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.795883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.795851 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9"] Nov 03 23:40:35.800275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.797676 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt8bm\" (UniqueName: \"kubernetes.io/projected/1824481f-5659-4932-ab0b-6726f418b3a0-kube-api-access-pt8bm\") pod \"maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.801199 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.801178 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"default-dockercfg-5kdv6\"" Nov 03 23:40:35.804173 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.804153 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"openshift-service-ca.crt\"" Nov 03 23:40:35.806899 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.806877 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"kube-root-ca.crt\"" Nov 03 23:40:35.831184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.831120 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"simulator-w2-5738a9-kserve-self-signed-certs\"" Nov 03 23:40:35.853439 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.853407 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:35.876535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.876492 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-tmp-dir\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.876688 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.876555 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfs2q\" (UniqueName: \"kubernetes.io/projected/a66e1c08-6769-477b-bb4a-1970e09e6627-kube-api-access-hfs2q\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.876758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.876722 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-home\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.876813 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.876777 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-model-cache\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.876861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.876846 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66e1c08-6769-477b-bb4a-1970e09e6627-tls-certs\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.876911 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.876897 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-dshm\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.892899 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.892873 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:35.978240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.978179 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-home\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.978421 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.978305 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-model-cache\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.978421 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.978376 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66e1c08-6769-477b-bb4a-1970e09e6627-tls-certs\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.978543 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.978447 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-dshm\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.978543 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.978499 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-tmp-dir\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.978644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.978547 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfs2q\" (UniqueName: \"kubernetes.io/projected/a66e1c08-6769-477b-bb4a-1970e09e6627-kube-api-access-hfs2q\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.979410 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.979379 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-home\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.979664 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.979642 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-model-cache\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.980709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.980675 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-tmp-dir\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.984377 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.984351 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66e1c08-6769-477b-bb4a-1970e09e6627-tls-certs\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.985749 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.985728 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-dshm\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:35.992865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:35.992806 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfs2q\" (UniqueName: \"kubernetes.io/projected/a66e1c08-6769-477b-bb4a-1970e09e6627-kube-api-access-hfs2q\") pod \"simulator-w2-5738a9-kserve-659975c56b-hxqc9\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:36.123189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.123111 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:36.178912 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.178878 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf"] Nov 03 23:40:36.186114 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:36.184623 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1824481f_5659_4932_ab0b_6726f418b3a0.slice/crio-cba9298945087976de74a8332f651cceacf4cbba3c6a4e7262105125f263929a WatchSource:0}: Error finding container cba9298945087976de74a8332f651cceacf4cbba3c6a4e7262105125f263929a: Status 404 returned error can't find the container with id cba9298945087976de74a8332f651cceacf4cbba3c6a4e7262105125f263929a Nov 03 23:40:36.403574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.403544 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9"] Nov 03 23:40:36.405140 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:36.405114 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda66e1c08_6769_477b_bb4a_1970e09e6627.slice/crio-f417c481bc6ba610ce9bd9e9355000b150da3342196515ec0d4ca98677eeaa6c WatchSource:0}: Error finding container f417c481bc6ba610ce9bd9e9355000b150da3342196515ec0d4ca98677eeaa6c: Status 404 returned error can't find the container with id f417c481bc6ba610ce9bd9e9355000b150da3342196515ec0d4ca98677eeaa6c Nov 03 23:40:36.557236 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.557203 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:36.979217 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.979180 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" event={"ID":"a66e1c08-6769-477b-bb4a-1970e09e6627","Type":"ContainerStarted","Data":"dfde2ba6a713e9f98db94b7c83af1ad098f927a0d6c2fd055d31069d435a75ac"} Nov 03 23:40:36.979648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.979232 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" event={"ID":"a66e1c08-6769-477b-bb4a-1970e09e6627","Type":"ContainerStarted","Data":"f417c481bc6ba610ce9bd9e9355000b150da3342196515ec0d4ca98677eeaa6c"} Nov 03 23:40:36.980795 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.980774 6208 generic.go:356] "Generic (PLEG): container finished" podID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerID="71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020" exitCode=0 Nov 03 23:40:36.980902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.980845 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" event={"ID":"adbdb3b7-b3c7-4908-8ced-a4185bafa586","Type":"ContainerDied","Data":"71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020"} Nov 03 23:40:36.981267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.981245 6208 scope.go:122] "RemoveContainer" containerID="71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020" Nov 03 23:40:36.982432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.982409 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" event={"ID":"1824481f-5659-4932-ab0b-6726f418b3a0","Type":"ContainerStarted","Data":"dc23a685b94afb0240a3653054a8aace10116a26ab254ec678d35f0465098cf3"} Nov 03 23:40:36.982524 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.982452 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" event={"ID":"1824481f-5659-4932-ab0b-6726f418b3a0","Type":"ContainerStarted","Data":"cba9298945087976de74a8332f651cceacf4cbba3c6a4e7262105125f263929a"} Nov 03 23:40:36.982524 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.982478 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:36.982601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.982569 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:36.982645 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.982631 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:36.982680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:36.982666 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:37.002670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:37.002634 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podStartSLOduration=2.002621004 podStartE2EDuration="2.002621004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:36.999817759 +0000 UTC m=+655.822413923" watchObservedRunningTime="2026-11-03 23:40:37.002621004 +0000 UTC m=+655.825217166" Nov 03 23:40:37.024117 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:37.024076 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" podStartSLOduration=1.589295227 podStartE2EDuration="2.0240627s" totalImagesPullingTime="434.767473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:37.021047414 +0000 UTC m=+655.843643574" watchObservedRunningTime="2026-11-03 23:40:37.0240627 +0000 UTC m=+655.846658861" Nov 03 23:40:37.124212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:37.124178 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:37.347356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:37.347269 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:37.990182 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:37.990149 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" event={"ID":"adbdb3b7-b3c7-4908-8ced-a4185bafa586","Type":"ContainerStarted","Data":"848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02"} Nov 03 23:40:38.307570 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:38.307486 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:38.307728 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:38.307710 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 23:40:38.307785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:38.307767 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 23:40:38.444080 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:38.444019 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:38.516577 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:38.516547 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:38.531627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:38.531594 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:38.563113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:38.553432 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:39.007649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:39.007621 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:40:39.084203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:39.084165 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:39.193049 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:39.193001 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:39.308113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:39.308015 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 23:40:39.308113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:39.308086 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 23:40:39.337344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:39.337308 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:40.308666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:40.308625 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 23:40:40.309121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:40.308702 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 23:40:40.600975 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:40.600882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:41.308004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:41.307968 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 23:40:41.308198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:41.308026 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 23:40:41.652403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:41.652303 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:42.307859 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:42.307817 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 23:40:42.308072 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:42.307875 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 23:40:42.442514 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:42.442474 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:42.682920 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:42.682846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:42.995767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:42.995739 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 03 23:40:43.313212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:43.313134 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:43.313500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:43.313477 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:43.314268 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:43.314250 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 03 23:40:43.422794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:43.422762 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:44.153216 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:44.153184 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:44.948685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:44.948644 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:45.291248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:45.291213 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:45.736928 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:45.736898 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:46.124131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:46.124049 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:46.136892 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:46.136868 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:46.320145 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:46.320113 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:46.651018 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:46.650988 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:47.053993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:47.053964 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:40:47.651708 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:47.651677 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:48.450579 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:48.450547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:49.329728 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:49.329696 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:50.254783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:50.254749 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:50.360608 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:50.360574 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:50.879466 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:50.879432 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:51.251536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:51.251504 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:51.625947 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:51.625814 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:51.734470 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:51.734431 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:52.245652 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.245623 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:52.388675 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.388642 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt"] Nov 03 23:40:52.394640 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.394619 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.403992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.403187 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-80f8ba-openshift-default-dockercfg-rx8cf\"" Nov 03 23:40:52.407081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.407057 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt"] Nov 03 23:40:52.454804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.454771 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2599\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-kube-api-access-p2599\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.454965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.454828 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b457bf-87e3-418f-bed9-8defb674e773-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.454965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.454941 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b457bf-87e3-418f-bed9-8defb674e773-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.455060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.454981 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.455110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.455095 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.455151 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.455129 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.455184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.455173 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.455226 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.455214 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.455262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.455239 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.545811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.545701 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:52.555901 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.555871 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b457bf-87e3-418f-bed9-8defb674e773-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.555921 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.555965 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.555993 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556022 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556101 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556136 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556197 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2599\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-kube-api-access-p2599\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556246 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b457bf-87e3-418f-bed9-8defb674e773-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556532 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556381 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556532 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556413 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556664 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556645 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b457bf-87e3-418f-bed9-8defb674e773-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556795 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556773 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.556849 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.556789 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.558351 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.558330 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.558611 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.558592 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b457bf-87e3-418f-bed9-8defb674e773-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.574671 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.574644 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.574800 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.574771 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2599\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-kube-api-access-p2599\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:52.710050 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:52.709998 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:53.037976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:53.037950 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt"] Nov 03 23:40:53.039614 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:40:53.039592 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b457bf_87e3_418f_bed9_8defb674e773.slice/crio-f943e2e6d6e93a1af9bb2e6feada274fbeea711b576135d7398827033b1c2ed5 WatchSource:0}: Error finding container f943e2e6d6e93a1af9bb2e6feada274fbeea711b576135d7398827033b1c2ed5: Status 404 returned error can't find the container with id f943e2e6d6e93a1af9bb2e6feada274fbeea711b576135d7398827033b1c2ed5 Nov 03 23:40:53.041850 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:53.041817 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:53.041942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:53.041877 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:53.041942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:53.041905 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:40:53.077270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:53.077239 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" event={"ID":"19b457bf-87e3-418f-bed9-8defb674e773","Type":"ContainerStarted","Data":"f943e2e6d6e93a1af9bb2e6feada274fbeea711b576135d7398827033b1c2ed5"} Nov 03 23:40:53.887922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:53.887876 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:54.086106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:54.086066 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" event={"ID":"19b457bf-87e3-418f-bed9-8defb674e773","Type":"ContainerStarted","Data":"7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450"} Nov 03 23:40:54.117838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:54.117790 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" podStartSLOduration=2.1177770320000002 podStartE2EDuration="2.117777032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:40:54.116983998 +0000 UTC m=+672.939580161" watchObservedRunningTime="2026-11-03 23:40:54.117777032 +0000 UTC m=+672.940373193" Nov 03 23:40:54.710805 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:54.710770 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:54.712483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:54.712457 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 23:40:54.712589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:54.712502 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" podUID="19b457bf-87e3-418f-bed9-8defb674e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 23:40:54.800089 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:54.800028 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:55.379504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:55.379468 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:55.710652 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:55.710619 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 23:40:55.710842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:55.710678 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" podUID="19b457bf-87e3-418f-bed9-8defb674e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 23:40:56.192748 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:56.192659 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:56.710750 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:56.710721 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 23:40:56.711204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:56.711136 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" podUID="19b457bf-87e3-418f-bed9-8defb674e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 23:40:57.578252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:57.578217 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:57.711251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:57.711221 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 23:40:57.711621 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:57.711273 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" podUID="19b457bf-87e3-418f-bed9-8defb674e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 23:40:58.577859 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:58.577823 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:58.710936 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:58.710904 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 23:40:58.711176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:58.711128 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" podUID="19b457bf-87e3-418f-bed9-8defb674e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 23:40:58.966666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:58.966625 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:59.626495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:59.626450 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:40:59.785408 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:59.785378 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:59.785711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:59.785680 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:40:59.786446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:40:59.786427 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:41:00.635905 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:00.635864 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:00.746898 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:00.746866 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:01.614336 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:01.614306 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:02.418541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:02.418508 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:03.485586 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:03.485546 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:04.326909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:04.326872 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:05.311463 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:05.311424 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:05.709598 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:41:05.709562 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:41:06.342427 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:06.342390 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:07.439898 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:07.439862 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:08.241407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:08.241374 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:08.862210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:08.862178 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr"] Nov 03 23:41:08.867602 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:08.867575 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:08.875316 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:08.875293 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"openshift-service-ca.crt\"" Nov 03 23:41:08.875782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:08.875733 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"default-dockercfg-fjmq9\"" Nov 03 23:41:08.875782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:08.875781 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"premium-w3-c60893-kserve-self-signed-certs\"" Nov 03 23:41:08.875935 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:08.875843 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"kube-root-ca.crt\"" Nov 03 23:41:08.879656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:08.879632 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr"] Nov 03 23:41:08.923343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:08.923303 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:09.051132 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.051093 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:09.064519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.064488 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a221959-932c-4943-b9ef-cf46b6e0d895-tls-certs\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.064670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.064554 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-home\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.064721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.064678 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-dshm\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.064756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.064719 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhpt\" (UniqueName: \"kubernetes.io/projected/8a221959-932c-4943-b9ef-cf46b6e0d895-kube-api-access-snhpt\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.064808 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.064793 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-tmp-dir\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.064846 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.064829 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-model-cache\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.166277 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.166202 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a221959-932c-4943-b9ef-cf46b6e0d895-tls-certs\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.166277 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.166270 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-home\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.166495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.166302 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-dshm\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.166495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.166319 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhpt\" (UniqueName: \"kubernetes.io/projected/8a221959-932c-4943-b9ef-cf46b6e0d895-kube-api-access-snhpt\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.166495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.166370 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-tmp-dir\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.166495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.166393 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-model-cache\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.166815 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.166787 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-tmp-dir\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.166963 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.166819 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-home\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.167091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.167072 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-model-cache\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.168681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.168662 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-dshm\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.168952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.168928 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a221959-932c-4943-b9ef-cf46b6e0d895-tls-certs\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.179010 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.178990 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhpt\" (UniqueName: \"kubernetes.io/projected/8a221959-932c-4943-b9ef-cf46b6e0d895-kube-api-access-snhpt\") pod \"premium-w3-c60893-kserve-6c9c78f87b-9glvr\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.183869 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.183849 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:09.251887 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.251855 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:09.351623 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.351587 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:09.460559 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.460535 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr"] Nov 03 23:41:09.462432 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:41:09.462403 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a221959_932c_4943_b9ef_cf46b6e0d895.slice/crio-50d04c85822bdb511df2a72a435710ef289d9b3ac9882dc227dc85326938b005 WatchSource:0}: Error finding container 50d04c85822bdb511df2a72a435710ef289d9b3ac9882dc227dc85326938b005: Status 404 returned error can't find the container with id 50d04c85822bdb511df2a72a435710ef289d9b3ac9882dc227dc85326938b005 Nov 03 23:41:09.951637 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:09.951548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:10.177323 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:10.177283 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" event={"ID":"8a221959-932c-4943-b9ef-cf46b6e0d895","Type":"ContainerStarted","Data":"e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512"} Nov 03 23:41:10.177480 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:10.177334 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" event={"ID":"8a221959-932c-4943-b9ef-cf46b6e0d895","Type":"ContainerStarted","Data":"50d04c85822bdb511df2a72a435710ef289d9b3ac9882dc227dc85326938b005"} Nov 03 23:41:10.184606 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:10.184580 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:10.234117 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:10.234070 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podStartSLOduration=2.234057275 podStartE2EDuration="2.234057275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:10.231923684 +0000 UTC m=+689.054519847" watchObservedRunningTime="2026-11-03 23:41:10.234057275 +0000 UTC m=+689.056653440" Nov 03 23:41:10.852739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:10.852690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:11.958384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:11.958352 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:13.052792 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:13.052759 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:13.939467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:13.939421 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:14.828704 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:14.828665 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:15.384760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:15.384725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:15.728574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:15.728535 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:16.679958 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:16.679920 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:17.531504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:17.531465 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:18.556962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:18.556925 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:19.184302 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:19.184271 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:19.197129 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:19.197105 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:19.256282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:19.256257 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:41:19.338237 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:19.338204 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:20.218694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:20.218659 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:20.941022 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:20.940981 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:21.681375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:21.681341 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:22.782555 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:22.782444 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:23.799618 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:23.799469 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:24.629343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:24.629305 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:25.498877 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:25.498843 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:26.256625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:26.256581 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:27.237024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:27.236968 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:28.083721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:28.083679 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:28.959404 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:28.959361 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:29.955261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:29.955224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:30.846351 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:30.846308 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:31.862473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:31.862436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:31.871725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:31.871694 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd"] Nov 03 23:41:31.875505 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:31.875487 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:31.880958 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:31.880935 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5cb349-openshift-default-dockercfg-qzhc9\"" Nov 03 23:41:31.893013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:31.892987 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd"] Nov 03 23:41:32.041861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.041271 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.041861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.041343 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.041861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.041379 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9z55\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-kube-api-access-v9z55\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.041861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.041425 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.041861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.041453 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.041861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.041495 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.041861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.041537 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.041861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.041595 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.041861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.041653 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.142558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142469 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.142558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142532 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.142806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142579 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.142806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142634 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.142806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142685 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.142806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142715 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9z55\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-kube-api-access-v9z55\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.142806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142762 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.142806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142784 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.143166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142848 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.143166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.142973 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.143166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.143017 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.143322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.143182 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.143605 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.143578 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.144412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.144377 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.145463 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.145438 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.146002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.145971 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.162438 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.162394 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9z55\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-kube-api-access-v9z55\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.162691 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.162672 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.190978 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.190644 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:32.525662 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.525621 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd"] Nov 03 23:41:32.532361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.532206 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:41:32.532361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.532291 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:41:32.532361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.532342 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:41:32.736583 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:32.736522 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:33.395595 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:33.395549 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" event={"ID":"02ef8451-9b7e-41ab-b0d4-02987d5f97d2","Type":"ContainerStarted","Data":"10e1529d128f8ca4595a5d657a3bd1546cea687458edeb23647ecec84f5d8595"} Nov 03 23:41:33.396339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:33.396307 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" event={"ID":"02ef8451-9b7e-41ab-b0d4-02987d5f97d2","Type":"ContainerStarted","Data":"e02cece774860095fd4df25ea5d0d79a28a6472bbc6dc327df44662382af8b3a"} Nov 03 23:41:33.452594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:33.452460 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" podStartSLOduration=2.452432825 podStartE2EDuration="2.452432825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:33.43611026 +0000 UTC m=+712.258706423" watchObservedRunningTime="2026-11-03 23:41:33.452432825 +0000 UTC m=+712.275028987" Nov 03 23:41:33.618199 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:33.617496 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:33.865064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:33.854819 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6"] Nov 03 23:41:33.865064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:33.863470 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:33.884363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:33.884329 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5738a9-openshift-default-dockercfg-ds6mb\"" Nov 03 23:41:33.887186 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:33.887100 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6"] Nov 03 23:41:34.017416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.017377 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.017416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.017418 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.017685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.017512 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.017685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.017626 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.017794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.017692 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-kube-api-access-6rktz\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.017794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.017720 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.017794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.017782 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.017925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.017852 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.017925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.017910 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.118806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.118711 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.118806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.118767 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119069 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.118834 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119069 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.118865 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119069 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.118921 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.119109 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.119146 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-kube-api-access-6rktz\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119434 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.119333 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119585 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.119536 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119585 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.119256 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.119631 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.119633 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.119637 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.119976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.119949 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.121384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.121349 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.121885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.121867 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.132493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.132471 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.132615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.132584 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-kube-api-access-6rktz\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.192510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.192471 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:34.194354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.194325 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 23:41:34.194446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.194379 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 23:41:34.196550 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.196532 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:34.519138 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.519019 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6"] Nov 03 23:41:34.521609 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:41:34.521575 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c279b6_0231_4d45_b0a8_8f05432e22ae.slice/crio-3dc5bb9a7e8cb814235bf407828ec64ddcc2cade00022e8697550f05851159bd WatchSource:0}: Error finding container 3dc5bb9a7e8cb814235bf407828ec64ddcc2cade00022e8697550f05851159bd: Status 404 returned error can't find the container with id 3dc5bb9a7e8cb814235bf407828ec64ddcc2cade00022e8697550f05851159bd Nov 03 23:41:34.524179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.524128 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:41:34.524303 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.524205 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:41:34.524303 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.524266 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:41:34.544102 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:34.544073 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:35.202060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:35.199154 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 23:41:35.202060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:35.199214 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 23:41:35.342581 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:35.342543 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:35.420068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:35.420018 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" event={"ID":"c6c279b6-0231-4d45-b0a8-8f05432e22ae","Type":"ContainerStarted","Data":"5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30"} Nov 03 23:41:35.420269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:35.420090 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" event={"ID":"c6c279b6-0231-4d45-b0a8-8f05432e22ae","Type":"ContainerStarted","Data":"3dc5bb9a7e8cb814235bf407828ec64ddcc2cade00022e8697550f05851159bd"} Nov 03 23:41:35.738169 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:41:35.738085 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:41:36.149780 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:36.149698 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:36.201059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:36.200694 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:36.201378 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:36.201060 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 23:41:36.201378 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:36.201108 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 23:41:36.213134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:36.212347 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 23:41:36.213134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:36.212411 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 23:41:36.950367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:36.950338 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:37.193055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:37.192449 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 23:41:37.193055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:37.192512 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 23:41:37.197389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:37.197358 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 23:41:37.197601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:37.197577 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 23:41:37.897056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:37.896583 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:38.192548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:38.192440 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 23:41:38.193147 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:38.193109 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 23:41:38.198390 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:38.198360 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 23:41:38.198600 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:38.198573 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 23:41:39.181966 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:39.181928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:39.192044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:39.191989 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 23:41:39.192246 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:39.192057 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 23:41:39.197384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:39.197350 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 23:41:39.199628 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:39.199551 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 23:41:40.188682 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:40.188645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:40.195640 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:40.195613 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:40.195978 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:40.195958 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:40.196684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:40.196668 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 03 23:41:40.196829 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:40.196810 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 23:41:40.196873 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:40.196853 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 23:41:40.222962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:40.222901 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" podStartSLOduration=7.222884788 podStartE2EDuration="7.222884788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:41:35.451663728 +0000 UTC m=+714.274259893" watchObservedRunningTime="2026-11-03 23:41:40.222884788 +0000 UTC m=+719.045480949" Nov 03 23:41:41.251458 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:41.251419 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:41.439790 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:41.439757 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:41.440879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:41.440857 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:41.442269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:41.442250 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 03 23:41:42.169482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:42.169449 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:43.332330 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:43.332291 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:44.331921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:44.331882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:45.140600 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:45.140563 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:45.355972 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:45.355930 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:45.983135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:45.983098 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:46.745380 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:46.745342 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:47.533868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:47.533814 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:48.327320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:48.327273 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:49.147431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:49.147391 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:50.059513 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:50.059429 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:51.168057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:51.167992 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:51.929861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:51.929828 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:52.861159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:52.861120 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:53.734915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:53.734879 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:54.552771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:54.552732 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:55.362910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:55.362857 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:55.463115 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:55.463068 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:56.151653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:56.151612 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:57.018382 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:57.018351 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:57.760525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:57.760494 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:57.861300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:57.861263 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:58.072711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:58.072629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:58.174098 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:58.174062 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:41:59.148442 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:41:59.148404 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:00.120971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:00.120936 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:00.328184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:00.328151 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:00.658061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:00.657151 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:42:00.658061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:00.657237 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:42:01.120936 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:01.120903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:01.939270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:01.939224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:02.823854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:02.823811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:03.737092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:03.729309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:04.738997 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:04.738956 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:05.750283 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:42:05.750248 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:42:06.110692 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:06.110601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:07.162534 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:07.162496 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:08.032807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:08.032770 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:08.751173 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:08.751138 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:09.557552 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:09.557512 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:10.344482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:10.344436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:11.266687 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:11.266647 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:12.041606 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:12.041567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:12.848078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:12.848009 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:13.703142 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:13.703102 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:14.162395 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:14.162337 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:14.292060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:14.288173 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:15.653241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:15.653191 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:42:15.653938 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:15.653890 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:42:15.736810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:15.736025 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:16.710773 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:16.710739 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:17.842009 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:17.841975 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:17.959684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:17.959654 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:18.877175 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:18.877144 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:18.959497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:18.959462 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp"] Nov 03 23:42:18.970771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:18.970723 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:18.981252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:18.981224 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp"] Nov 03 23:42:19.101718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.101659 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjnx6\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-kube-api-access-hjnx6\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.101898 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.101804 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.101898 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.101872 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.102019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.101912 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.102087 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.102024 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.102134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.102120 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.102182 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.102149 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.102182 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.102170 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.102313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.102292 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.203973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.203883 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.203973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.203936 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.203973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.203968 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204291 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.203988 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204291 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204014 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204291 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204081 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204291 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204182 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjnx6\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-kube-api-access-hjnx6\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204513 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204304 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204513 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204341 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204513 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204397 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204513 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204461 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204726 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204577 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.204838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204816 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.205013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.204992 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.206288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.206257 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.206661 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.206641 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.216466 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.216435 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.216671 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.216648 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjnx6\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-kube-api-access-hjnx6\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.291682 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.291641 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:19.578797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.578767 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp"] Nov 03 23:42:19.580693 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:42:19.580660 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ddc9ee_841e_4f8c_a4b8_8343b4c470f7.slice/crio-711bf00d8b4f68fd5116a27a39202e67adbdb355009941721ca903ea2b9fa622 WatchSource:0}: Error finding container 711bf00d8b4f68fd5116a27a39202e67adbdb355009941721ca903ea2b9fa622: Status 404 returned error can't find the container with id 711bf00d8b4f68fd5116a27a39202e67adbdb355009941721ca903ea2b9fa622 Nov 03 23:42:19.583177 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.583139 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:42:19.583300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.583210 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:42:19.583300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.583249 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:42:19.865761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.865659 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" event={"ID":"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7","Type":"ContainerStarted","Data":"c4c957e5968d8146c5259bb227b383825bcd35f340adf73c27bcbd11d5dd9799"} Nov 03 23:42:19.865761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.865711 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" event={"ID":"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7","Type":"ContainerStarted","Data":"711bf00d8b4f68fd5116a27a39202e67adbdb355009941721ca903ea2b9fa622"} Nov 03 23:42:19.871726 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.871693 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:19.892864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:19.892815 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" podStartSLOduration=1.892800934 podStartE2EDuration="1.892800934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:19.889691493 +0000 UTC m=+758.712287654" watchObservedRunningTime="2026-11-03 23:42:19.892800934 +0000 UTC m=+758.715397096" Nov 03 23:42:20.297057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:20.296194 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:20.297057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:20.296346 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 23:42:20.297057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:20.296392 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 23:42:21.198626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:21.198576 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:21.307056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:21.306259 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 23:42:21.307056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:21.306322 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 23:42:22.293057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:22.292396 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 23:42:22.293057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:22.292466 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 23:42:22.459058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:22.458919 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:23.369056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:23.368646 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 23:42:23.369056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:23.368717 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 23:42:23.786305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:23.786273 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:24.292384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:24.292346 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 23:42:24.292563 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:24.292405 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 23:42:24.878101 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:24.876410 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:25.220905 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.220861 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd"] Nov 03 23:42:25.224981 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.224960 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.228951 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.228927 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-dshm\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.229163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.228970 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-tmp-dir\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.229163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.229005 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5k5\" (UniqueName: \"kubernetes.io/projected/e200da9d-3c7d-4904-9d6d-88badac0a185-kube-api-access-6x5k5\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.229163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.229068 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e200da9d-3c7d-4904-9d6d-88badac0a185-tls-certs\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.229163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.229131 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-model-cache\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.229431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.229212 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-home\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.236116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.235449 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5738a9\"/\"distinct-w2-5738a9-kserve-self-signed-certs\"" Nov 03 23:42:25.277063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.272114 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:25.279157 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.279134 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd"] Nov 03 23:42:25.292854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.292824 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 23:42:25.292964 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.292892 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 23:42:25.330229 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.330196 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-home\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.330364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.330284 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-dshm\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.330364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.330320 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-tmp-dir\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.330364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.330339 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5k5\" (UniqueName: \"kubernetes.io/projected/e200da9d-3c7d-4904-9d6d-88badac0a185-kube-api-access-6x5k5\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.330525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.330374 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e200da9d-3c7d-4904-9d6d-88badac0a185-tls-certs\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.330525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.330472 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-model-cache\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.330724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.330695 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-tmp-dir\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.330813 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.330727 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-home\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.330868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.330832 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-model-cache\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.332729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.332703 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-dshm\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.332838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.332820 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e200da9d-3c7d-4904-9d6d-88badac0a185-tls-certs\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.347362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.347331 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5k5\" (UniqueName: \"kubernetes.io/projected/e200da9d-3c7d-4904-9d6d-88badac0a185-kube-api-access-6x5k5\") pod \"distinct-w2-5738a9-kserve-59c866dd6-l4zxd\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.549564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.549475 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:25.850076 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.850051 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd"] Nov 03 23:42:25.852163 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:42:25.852137 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode200da9d_3c7d_4904_9d6d_88badac0a185.slice/crio-f5f9a9439bba2fe23f1b508766d25295c044cd801c0a808ca53e3e3fc671d934 WatchSource:0}: Error finding container f5f9a9439bba2fe23f1b508766d25295c044cd801c0a808ca53e3e3fc671d934: Status 404 returned error can't find the container with id f5f9a9439bba2fe23f1b508766d25295c044cd801c0a808ca53e3e3fc671d934 Nov 03 23:42:25.976859 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:25.976832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:26.012927 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:26.012894 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" event={"ID":"e200da9d-3c7d-4904-9d6d-88badac0a185","Type":"ContainerStarted","Data":"fd51b633090c1c0ba023535c2b35ab398d784609cb403c6298a33e91691e2723"} Nov 03 23:42:26.012927 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:26.012936 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" event={"ID":"e200da9d-3c7d-4904-9d6d-88badac0a185","Type":"ContainerStarted","Data":"f5f9a9439bba2fe23f1b508766d25295c044cd801c0a808ca53e3e3fc671d934"} Nov 03 23:42:26.038271 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:26.038204 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podStartSLOduration=1.038182336 podStartE2EDuration="1.038182336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:26.034970242 +0000 UTC m=+764.857566417" watchObservedRunningTime="2026-11-03 23:42:26.038182336 +0000 UTC m=+764.860778500" Nov 03 23:42:26.292479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:26.292418 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 23:42:26.292675 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:26.292486 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 23:42:26.550189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:26.550097 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:27.135043 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:27.135000 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:27.296838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:27.296809 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:27.297259 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:27.297242 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:27.297848 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:27.297825 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:42:27.952984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:27.952950 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:28.752238 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:28.752192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:29.542659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:29.542626 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:30.446585 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:30.446548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:31.412159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:31.412124 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:33.082095 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:33.080635 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:34.493983 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:34.493891 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:35.550594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:35.550555 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:35.563385 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:35.563356 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:35.704807 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:42:35.704763 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:42:35.810874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:35.810784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:36.212649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:36.212388 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:42:36.954983 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:36.954943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:37.649053 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:37.644026 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:37.779059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:37.775172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:38.107452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:38.107413 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:39.031922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:39.031868 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:40.079384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:40.079309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:40.974313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:40.974276 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:41.798668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:41.798634 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:42.124780 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:42.124672 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:42.949404 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:42.948467 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:43.728316 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:43.728283 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:44.697339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:44.697298 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:45.433354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:45.433317 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:45.635192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:45.635152 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:46.832219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:46.832181 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:47.962623 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:47.962486 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:48.841338 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:48.841305 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:49.636866 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:49.636827 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:50.447000 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:50.446964 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:51.175418 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.175380 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95"] Nov 03 23:42:51.180577 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.180556 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.191408 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.191385 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c60893\"/\"distinct-w3-c60893-kserve-self-signed-certs\"" Nov 03 23:42:51.193230 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.193205 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95"] Nov 03 23:42:51.227290 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.227253 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:51.284006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.283979 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tls-certs\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.284184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.284023 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tmp-dir\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.284227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.284195 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rlrf\" (UniqueName: \"kubernetes.io/projected/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-kube-api-access-7rlrf\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.284262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.284254 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-home\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.284309 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.284295 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-model-cache\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.284356 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.284345 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-dshm\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.332394 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.332328 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:51.385348 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.385312 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rlrf\" (UniqueName: \"kubernetes.io/projected/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-kube-api-access-7rlrf\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.385535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.385388 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-home\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.385535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.385413 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-model-cache\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.385535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.385450 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-dshm\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.385535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.385510 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tls-certs\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.385749 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.385557 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tmp-dir\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.385889 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.385859 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-home\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.385889 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.385878 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-model-cache\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.386068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.385959 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tmp-dir\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.387755 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.387736 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-dshm\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.388024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.388005 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tls-certs\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.399007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.398984 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rlrf\" (UniqueName: \"kubernetes.io/projected/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-kube-api-access-7rlrf\") pod \"distinct-w3-c60893-kserve-78fd879db7-7tb95\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.492612 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.492576 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:51.771772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:51.771746 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95"] Nov 03 23:42:51.773792 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:42:51.773766 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6c6643f_325e_491e_a7cf_6ea8357bfe5b.slice/crio-a697b7181e7bda9a32beb59c0b45902a6fbaf814a279cb70ccd2968681ea4820 WatchSource:0}: Error finding container a697b7181e7bda9a32beb59c0b45902a6fbaf814a279cb70ccd2968681ea4820: Status 404 returned error can't find the container with id a697b7181e7bda9a32beb59c0b45902a6fbaf814a279cb70ccd2968681ea4820 Nov 03 23:42:52.194064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.193948 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:52.344057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.343462 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" event={"ID":"e6c6643f-325e-491e-a7cf-6ea8357bfe5b","Type":"ContainerStarted","Data":"6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0"} Nov 03 23:42:52.344057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.343505 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" event={"ID":"e6c6643f-325e-491e-a7cf-6ea8357bfe5b","Type":"ContainerStarted","Data":"a697b7181e7bda9a32beb59c0b45902a6fbaf814a279cb70ccd2968681ea4820"} Nov 03 23:42:52.364980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.364917 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podStartSLOduration=1.3648959139999999 podStartE2EDuration="1.364895914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:52.3646837 +0000 UTC m=+791.187279863" watchObservedRunningTime="2026-11-03 23:42:52.364895914 +0000 UTC m=+791.187492078" Nov 03 23:42:52.493742 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.493701 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:42:52.672470 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.672434 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j"] Nov 03 23:42:52.678084 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.678055 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.689780 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.689645 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j"] Nov 03 23:42:52.706174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.705509 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.706174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.705566 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.706174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.705659 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4f5p\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-kube-api-access-k4f5p\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.706174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.705700 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.706174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.705795 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.706174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.705836 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.706174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.705863 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.706174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.705950 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.706174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.705988 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.807435 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807343 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.807435 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807404 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.807435 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807438 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.807732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807459 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4f5p\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-kube-api-access-k4f5p\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.807732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807478 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.807732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807566 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.807732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807611 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.807732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807640 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.807732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807731 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.808054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.807921 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.808168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.808144 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.808553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.808522 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.808553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.808539 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.808763 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.808737 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.810226 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.810203 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.810314 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.810262 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.820500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.820474 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:52.821002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:52.820976 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4f5p\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-kube-api-access-k4f5p\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:53.006062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:53.005387 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:53.342209 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:53.342175 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j"] Nov 03 23:42:53.343115 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:42:53.343088 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e6bc29_8cd4_4910_9a12_3686b55c19b4.slice/crio-455ad7903d129880cafcdda41d7d609ceb253db2e03a1034b18866ded51dee28 WatchSource:0}: Error finding container 455ad7903d129880cafcdda41d7d609ceb253db2e03a1034b18866ded51dee28: Status 404 returned error can't find the container with id 455ad7903d129880cafcdda41d7d609ceb253db2e03a1034b18866ded51dee28 Nov 03 23:42:53.347819 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:53.347781 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:42:53.348023 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:53.347997 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:42:53.348279 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:53.348174 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:42:53.355889 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:53.355826 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" event={"ID":"f6e6bc29-8cd4-4910-9a12-3686b55c19b4","Type":"ContainerStarted","Data":"455ad7903d129880cafcdda41d7d609ceb253db2e03a1034b18866ded51dee28"} Nov 03 23:42:53.413676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:53.413646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:54.048838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:54.048801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:54.393103 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:54.392965 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" event={"ID":"f6e6bc29-8cd4-4910-9a12-3686b55c19b4","Type":"ContainerStarted","Data":"59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb"} Nov 03 23:42:54.420088 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:54.420001 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" podStartSLOduration=2.419984132 podStartE2EDuration="2.419984132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:42:54.419210059 +0000 UTC m=+793.241806246" watchObservedRunningTime="2026-11-03 23:42:54.419984132 +0000 UTC m=+793.242580296" Nov 03 23:42:54.837912 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:54.837873 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:55.007714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:55.007683 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:42:55.009519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:55.009490 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 23:42:55.009638 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:55.009554 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 23:42:55.736334 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:55.736300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:56.006915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:56.006828 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 23:42:56.006915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:56.006896 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 23:42:56.518965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:56.518930 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:57.006768 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:57.006734 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 23:42:57.007310 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:57.006800 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 23:42:57.372645 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:57.372549 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:58.006685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:58.006647 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 23:42:58.006857 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:58.006711 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 23:42:58.146864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:58.146812 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:58.922420 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:58.922368 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:42:59.007319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:59.007288 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 23:42:59.007753 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:59.007357 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 23:42:59.771847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:42:59.771809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:00.007239 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:00.007166 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 23:43:00.007239 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:00.007237 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 23:43:00.395083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:00.395046 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:00.761496 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:00.761462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:01.493742 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:01.493703 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:43:01.514336 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:01.514295 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:43:01.646273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:01.646242 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:02.048242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:02.048198 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:43:02.048465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:02.048273 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:43:02.093771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:02.093734 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:43:02.094393 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:02.094363 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:43:02.095398 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:02.095375 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 03 23:43:02.461350 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:02.461323 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:43:02.555741 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:02.555690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:03.334689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:03.334653 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:04.435482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:04.435442 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:05.359467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:05.359433 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:05.490804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:05.490763 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:05.705132 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:43:05.705045 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:43:06.152052 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:06.151997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:07.052922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:07.052891 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:07.740518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:07.740487 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:08.145339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:08.145256 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:09.049197 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:09.049166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:09.841311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:09.841273 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:10.756803 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:10.756763 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:11.041113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:11.041010 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:11.145387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:11.145349 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:11.699352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:11.699320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:12.565306 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:12.565278 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:13.420057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:13.419653 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:13.588969 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:13.588914 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:14.554688 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:14.554646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:15.415062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.412160 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp"] Nov 03 23:43:15.447982 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.447750 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp"] Nov 03 23:43:15.447982 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.447873 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.502809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.502769 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:15.585127 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.585086 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzrz\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-kube-api-access-9kzrz\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.585300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.585164 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.585358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.585316 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.585450 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.585423 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.585506 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.585467 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7224-8eef-4621-b360-39b48ebee08e-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.585684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.585652 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.585761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.585729 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7224-8eef-4621-b360-39b48ebee08e-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.585802 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.585772 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.585890 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.585871 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.686819 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.686721 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.686819 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.686802 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687104 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.686839 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7224-8eef-4621-b360-39b48ebee08e-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687104 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.686905 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687104 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.686944 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7224-8eef-4621-b360-39b48ebee08e-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687104 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.686976 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687104 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.687096 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.687165 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzrz\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-kube-api-access-9kzrz\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.687193 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.687210 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687543 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.687390 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687591 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.687488 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687591 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.687515 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.687865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.687849 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7224-8eef-4621-b360-39b48ebee08e-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.689511 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.689481 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7224-8eef-4621-b360-39b48ebee08e-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.689636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.689538 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.700191 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.700167 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.700638 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.700613 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzrz\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-kube-api-access-9kzrz\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:15.802799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:15.802759 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:16.112760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:16.112734 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp"] Nov 03 23:43:16.114195 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:43:16.114171 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27b7224_8eef_4621_b360_39b48ebee08e.slice/crio-ed8f9ad3bb93de27cac7932f37046b721fbf3f25f107dc8a848bcb8ec999f155 WatchSource:0}: Error finding container ed8f9ad3bb93de27cac7932f37046b721fbf3f25f107dc8a848bcb8ec999f155: Status 404 returned error can't find the container with id ed8f9ad3bb93de27cac7932f37046b721fbf3f25f107dc8a848bcb8ec999f155 Nov 03 23:43:16.116630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:16.116597 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:43:16.116701 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:16.116663 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:43:16.116738 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:16.116705 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:43:16.438875 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:16.438768 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:16.539764 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:16.539728 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" event={"ID":"e27b7224-8eef-4621-b360-39b48ebee08e","Type":"ContainerStarted","Data":"62d3c92739535121c979fda9adc3fa0cfaf26467014929bb2f927533baa7cad0"} Nov 03 23:43:16.539938 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:16.539779 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" event={"ID":"e27b7224-8eef-4621-b360-39b48ebee08e","Type":"ContainerStarted","Data":"ed8f9ad3bb93de27cac7932f37046b721fbf3f25f107dc8a848bcb8ec999f155"} Nov 03 23:43:16.565908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:16.565857 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" podStartSLOduration=1.5658448900000002 podStartE2EDuration="1.56584489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:16.563602722 +0000 UTC m=+815.386198920" watchObservedRunningTime="2026-11-03 23:43:16.56584489 +0000 UTC m=+815.388441052" Nov 03 23:43:16.803715 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:16.803672 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:17.379126 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:17.379085 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:17.804796 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:17.804765 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 23:43:17.805276 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:17.804816 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 23:43:18.379542 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:18.379496 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:18.803932 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:18.803894 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 23:43:18.804153 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:18.803948 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 23:43:19.108352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.108255 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh"] Nov 03 23:43:19.112165 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.112145 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.125093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.125066 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.125286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.125256 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.125411 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.125372 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.125473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.125418 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.125473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.125465 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srm9g\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-kube-api-access-srm9g\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.125588 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.125500 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.125588 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.125569 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.125693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.125612 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.125693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.125689 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.130831 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.130799 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh"] Nov 03 23:43:19.227173 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227133 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.227363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227186 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.227363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227253 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.227363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227289 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.227363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227317 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.227589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227428 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.227695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227495 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.227817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227761 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.227817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227775 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm9g\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-kube-api-access-srm9g\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.227945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227853 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.228004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.227950 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.228321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.228286 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.228405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.228299 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.228597 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.228578 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.229496 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.229438 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.229725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.229704 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.242473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.242444 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.242576 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.242553 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm9g\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-kube-api-access-srm9g\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.379668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.379557 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:19.431339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.431303 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:19.782545 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:43:19.782495 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b01bcf_1e5c_4587_8a75_f9734a590e20.slice/crio-adf3bb401c0101808d87617654176b6240b65f5234bff9a7b52e517059973703 WatchSource:0}: Error finding container adf3bb401c0101808d87617654176b6240b65f5234bff9a7b52e517059973703: Status 404 returned error can't find the container with id adf3bb401c0101808d87617654176b6240b65f5234bff9a7b52e517059973703 Nov 03 23:43:19.782883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.782860 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh"] Nov 03 23:43:19.785137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.785108 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:43:19.785228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.785168 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:43:19.785228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.785195 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:43:19.804696 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.804666 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 23:43:19.804816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:19.804733 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 23:43:20.388056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:20.375210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:20.608680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:20.608643 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" event={"ID":"d6b01bcf-1e5c-4587-8a75-f9734a590e20","Type":"ContainerStarted","Data":"f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f"} Nov 03 23:43:20.608890 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:20.608710 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" event={"ID":"d6b01bcf-1e5c-4587-8a75-f9734a590e20","Type":"ContainerStarted","Data":"adf3bb401c0101808d87617654176b6240b65f5234bff9a7b52e517059973703"} Nov 03 23:43:20.635435 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:20.635383 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" podStartSLOduration=1.635370647 podStartE2EDuration="1.635370647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:20.633695919 +0000 UTC m=+819.456292083" watchObservedRunningTime="2026-11-03 23:43:20.635370647 +0000 UTC m=+819.457966880" Nov 03 23:43:20.803457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:20.803412 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 23:43:20.803652 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:20.803478 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 23:43:21.404061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:21.395699 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:21.434318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:21.432662 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:21.434318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:21.433663 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 23:43:21.434318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:21.433711 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 23:43:21.506798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:21.506678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:21.804057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:21.803598 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 23:43:21.804057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:21.803685 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 23:43:22.432353 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:22.432321 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 23:43:22.432738 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:22.432387 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 23:43:22.803464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:22.803422 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 23:43:22.803662 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:22.803491 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 23:43:22.808421 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:22.808394 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:23.432563 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:23.432525 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 23:43:23.433078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:23.432588 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 23:43:23.821953 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:23.821923 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:23.822298 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:23.822263 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:23.823093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:23.823073 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 03 23:43:24.079142 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:24.079057 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:24.432359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:24.432278 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 23:43:24.432359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:24.432327 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 23:43:25.404599 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:25.404560 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:25.431980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:25.431939 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 23:43:25.432211 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:25.432001 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 23:43:26.432073 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:26.432021 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 23:43:26.432472 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:26.432133 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 23:43:26.697855 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:26.697753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:27.436801 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:27.436771 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:27.437230 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:27.437120 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:27.437779 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:27.437757 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 03 23:43:27.587027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:27.586971 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:28.707190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:28.707157 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:29.965866 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:29.965828 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:32.019158 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:32.019112 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:43:32.056388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:32.055253 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:43:32.196200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:32.196154 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 23:43:32.627842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:32.627810 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:33.346712 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:33.346675 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:34.808719 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:34.808679 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:34.954015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:34.953952 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:35.085251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:35.085166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:35.704537 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:43:35.704498 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:43:35.839861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:35.839817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:35.991572 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:35.991538 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:36.747938 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:36.747905 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:37.679028 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:37.678975 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:38.553245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:38.553205 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:39.483522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:39.483489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:40.361254 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:40.361218 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:41.354221 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:41.354189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:42.150806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:42.150753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:43.033400 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:43.033359 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:43.994944 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:43.994916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:44.901142 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:44.901099 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:46.570270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:46.570204 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:43:46.768055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:46.768001 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:47.498064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:47.497892 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:48.145967 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:48.145926 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:49.061229 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.061194 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:49.210096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.210029 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79"] Nov 03 23:43:49.215845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.215822 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.242162 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.242127 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79"] Nov 03 23:43:49.268912 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.268876 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.269123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.269100 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.269187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.269155 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.269297 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.269271 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.269364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.269320 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.269445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.269421 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.269500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.269465 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.269500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.269492 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.269615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.269521 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcpt\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-kube-api-access-qmcpt\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.370968 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.370885 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.370972 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371001 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371081 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371112 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371206 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371227 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371244 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371260 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcpt\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-kube-api-access-qmcpt\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371545 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-certs\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371571 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.371792 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371760 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-data\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.372002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.371977 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-credential-socket\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.372113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.372008 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istiod-ca-cert\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.373431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.373411 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-envoy\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.373568 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.373552 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-podinfo\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.384790 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.384767 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-token\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.390123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.390100 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcpt\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-kube-api-access-qmcpt\") pod \"e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.530687 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.530654 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:49.822760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.822728 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79"] Nov 03 23:43:49.825637 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:43:49.825600 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ed126e_cb5d_4fa5_aaf2_1cc63651849f.slice/crio-62150aeccabfda95329c94497fa9ddffd62303b198d36e601e393f2779a1f4e8 WatchSource:0}: Error finding container 62150aeccabfda95329c94497fa9ddffd62303b198d36e601e393f2779a1f4e8: Status 404 returned error can't find the container with id 62150aeccabfda95329c94497fa9ddffd62303b198d36e601e393f2779a1f4e8 Nov 03 23:43:49.828901 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.828864 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:43:49.829114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.829089 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:43:49.829268 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:49.829138 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:43:50.028673 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:50.028632 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" event={"ID":"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f","Type":"ContainerStarted","Data":"34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0"} Nov 03 23:43:50.028869 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:50.028686 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" event={"ID":"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f","Type":"ContainerStarted","Data":"62150aeccabfda95329c94497fa9ddffd62303b198d36e601e393f2779a1f4e8"} Nov 03 23:43:50.059059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:50.058990 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" podStartSLOduration=1.058977177 podStartE2EDuration="1.058977177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:43:50.058627274 +0000 UTC m=+848.881223464" watchObservedRunningTime="2026-11-03 23:43:50.058977177 +0000 UTC m=+848.881573338" Nov 03 23:43:50.229045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:50.229000 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:50.530971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:50.530880 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:50.532816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:50.532789 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 23:43:50.532946 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:50.532854 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 23:43:51.159681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:51.159648 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:51.531313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:51.531279 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 23:43:51.531706 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:51.531328 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 23:43:52.084350 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:52.084318 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:52.532004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:52.531965 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 23:43:52.532529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:52.532022 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 23:43:52.979663 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:52.979622 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:53.531876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:53.531837 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 23:43:53.532096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:53.531892 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 23:43:53.982897 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:53.982863 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:54.531021 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:54.530984 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 23:43:54.531218 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:54.531068 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 23:43:54.880883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:54.880794 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:55.531187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:55.531152 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 23:43:55.531373 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:55.531204 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 23:43:55.563055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:55.561350 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:55.986514 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:55.986478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:56.531541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:56.531505 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 23:43:56.531711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:56.531569 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 23:43:57.033864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:57.033825 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:57.536066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:57.536019 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:57.536490 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:57.536473 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:57.537284 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:57.537267 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 03 23:43:57.974104 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:57.974012 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:58.797735 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:58.797702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:43:59.644617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:43:59.644581 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:00.421660 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.421626 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb"] Nov 03 23:44:00.425313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.425289 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.430488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.430463 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"distinct-2-w1-5cb349-kserve-self-signed-certs\"" Nov 03 23:44:00.432772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.432751 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"default-dockercfg-6tjp2\"" Nov 03 23:44:00.433588 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.433221 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"openshift-service-ca.crt\"" Nov 03 23:44:00.433588 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.433323 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5cb349\"/\"kube-root-ca.crt\"" Nov 03 23:44:00.438013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.437987 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb"] Nov 03 23:44:00.479476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.479442 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:00.512830 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.512781 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwn69\" (UniqueName: \"kubernetes.io/projected/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-kube-api-access-gwn69\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.513027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.512925 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tls-certs\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.513129 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.513105 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-home\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.513176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.513154 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-dshm\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.513208 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.513187 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tmp-dir\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.513309 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.513290 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-model-cache\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.523352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.523315 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl"] Nov 03 23:44:00.527444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.527418 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.596130 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.596070 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl"] Nov 03 23:44:00.614610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.614566 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-home\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.614809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.614627 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-dshm\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.614809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.614669 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tmp-dir\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.614809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.614706 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.614809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.614760 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-model-cache\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.614809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.614793 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.615110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.614821 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.615110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.614865 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwn69\" (UniqueName: \"kubernetes.io/projected/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-kube-api-access-gwn69\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.615110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.614952 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tls-certs\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.615110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.615018 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.615110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.615078 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.615366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.615112 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.615366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.615143 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdrd\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-kube-api-access-sxdrd\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.615366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.615312 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.615366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.615342 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.615772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.615751 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-home\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.616339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.616315 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-model-cache\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.616662 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.616636 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tmp-dir\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.619305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.619273 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-dshm\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.619698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.619674 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tls-certs\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.629978 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.629946 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwn69\" (UniqueName: \"kubernetes.io/projected/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-kube-api-access-gwn69\") pod \"distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.630396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.630364 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:00.716565 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716533 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.716731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716575 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.716731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716594 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.716731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716617 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdrd\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-kube-api-access-sxdrd\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.716731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716671 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.716731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716687 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.717014 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716748 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.717014 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716813 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.717014 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716843 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.717014 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.716990 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.717208 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.717080 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.717253 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.717200 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.717338 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.717320 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.717588 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.717557 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.719264 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.719243 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.719387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.719369 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.730215 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.730193 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.730368 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.730351 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdrd\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-kube-api-access-sxdrd\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:00.740231 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.740208 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:00.843232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:00.843192 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:01.034650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.034617 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb"] Nov 03 23:44:01.035085 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:44:01.035061 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46d2d360_21ba_4c73_b6a1_fe28276a6f2f.slice/crio-06d01716ff729596fa3c21bef916b4bb57997c7e17d465f1c4adf15fbd039bf8 WatchSource:0}: Error finding container 06d01716ff729596fa3c21bef916b4bb57997c7e17d465f1c4adf15fbd039bf8: Status 404 returned error can't find the container with id 06d01716ff729596fa3c21bef916b4bb57997c7e17d465f1c4adf15fbd039bf8 Nov 03 23:44:01.126496 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.126454 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" event={"ID":"46d2d360-21ba-4c73-b6a1-fe28276a6f2f","Type":"ContainerStarted","Data":"31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d"} Nov 03 23:44:01.126666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.126509 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" event={"ID":"46d2d360-21ba-4c73-b6a1-fe28276a6f2f","Type":"ContainerStarted","Data":"06d01716ff729596fa3c21bef916b4bb57997c7e17d465f1c4adf15fbd039bf8"} Nov 03 23:44:01.142298 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.142146 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl"] Nov 03 23:44:01.145205 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:44:01.145167 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fba9ea6_4d1f_46e6_bf0f_6dcafe283642.slice/crio-ee9012c4f70e8eaf4eebc8564ccc2076c20d8472c19389ab313bfc2037822cab WatchSource:0}: Error finding container ee9012c4f70e8eaf4eebc8564ccc2076c20d8472c19389ab313bfc2037822cab: Status 404 returned error can't find the container with id ee9012c4f70e8eaf4eebc8564ccc2076c20d8472c19389ab313bfc2037822cab Nov 03 23:44:01.147906 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.147868 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:01.147982 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.147940 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:01.147982 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.147974 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:01.160388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.160335 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podStartSLOduration=1.160318389 podStartE2EDuration="1.160318389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:01.157247078 +0000 UTC m=+859.979843242" watchObservedRunningTime="2026-11-03 23:44:01.160318389 +0000 UTC m=+859.982914551" Nov 03 23:44:01.621337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.621294 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:01.741052 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:01.740662 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:02.181127 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.180408 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" event={"ID":"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642","Type":"ContainerStarted","Data":"c974345fc6e21bf1dbbb62bc345fc24522bd0305f7478255cafd2832facb89cc"} Nov 03 23:44:02.181127 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.180521 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" event={"ID":"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642","Type":"ContainerStarted","Data":"ee9012c4f70e8eaf4eebc8564ccc2076c20d8472c19389ab313bfc2037822cab"} Nov 03 23:44:02.219059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.218512 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" podStartSLOduration=2.218494591 podStartE2EDuration="2.218494591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:02.218425393 +0000 UTC m=+861.041021556" watchObservedRunningTime="2026-11-03 23:44:02.218494591 +0000 UTC m=+861.041090754" Nov 03 23:44:02.592060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.588401 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94"] Nov 03 23:44:02.616059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.614448 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.734010 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.733910 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94"] Nov 03 23:44:02.764057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.760792 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.785059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.781200 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.789154 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.789119 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31de50e-34be-4e03-8c70-e98d3d4cb577-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.793058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.789404 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnm5s\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-kube-api-access-vnm5s\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.799233 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.799189 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.799434 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.799414 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.807243 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.807220 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.835087 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.811185 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.835445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.835416 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.847233 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.847092 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:02.931060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.908169 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 23:44:02.931060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.908264 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 23:44:02.949091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.937831 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.949091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.937903 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31de50e-34be-4e03-8c70-e98d3d4cb577-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.949091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.937974 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnm5s\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-kube-api-access-vnm5s\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.949091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.938003 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.949091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.938027 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.949091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.938141 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.949091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.938224 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.949091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.938265 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:02.949091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.938310 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.000060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.952593 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.000060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.952945 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.000060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.953671 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31de50e-34be-4e03-8c70-e98d3d4cb577-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.000060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.959442 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.000060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:02.959707 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.103130 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.099596 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.103130 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.100337 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.109656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.109582 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.148688 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.139287 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnm5s\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-kube-api-access-vnm5s\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.220423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.220318 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:03.415393 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.413286 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:03.844522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.844489 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 23:44:03.844966 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.844555 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 23:44:03.979431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.978904 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94"] Nov 03 23:44:03.982673 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:44:03.982634 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31de50e_34be_4e03_8c70_e98d3d4cb577.slice/crio-2ad43171f96518ebabb4a6f2aa93eac3e8fa3e6c15458925d65fe590af4a7b54 WatchSource:0}: Error finding container 2ad43171f96518ebabb4a6f2aa93eac3e8fa3e6c15458925d65fe590af4a7b54: Status 404 returned error can't find the container with id 2ad43171f96518ebabb4a6f2aa93eac3e8fa3e6c15458925d65fe590af4a7b54 Nov 03 23:44:03.985249 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.985215 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:03.985337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.985283 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:03.985337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:03.985312 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:04.205667 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:04.205429 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" event={"ID":"a31de50e-34be-4e03-8c70-e98d3d4cb577","Type":"ContainerStarted","Data":"89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e"} Nov 03 23:44:04.205667 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:04.205495 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" event={"ID":"a31de50e-34be-4e03-8c70-e98d3d4cb577","Type":"ContainerStarted","Data":"2ad43171f96518ebabb4a6f2aa93eac3e8fa3e6c15458925d65fe590af4a7b54"} Nov 03 23:44:04.233275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:04.233166 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podStartSLOduration=2.233150214 podStartE2EDuration="2.233150214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:04.228931724 +0000 UTC m=+863.051527886" watchObservedRunningTime="2026-11-03 23:44:04.233150214 +0000 UTC m=+863.055746375" Nov 03 23:44:04.415838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:04.415801 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:04.619647 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:04.619555 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:04.844333 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:04.844296 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 23:44:04.844510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:04.844361 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 23:44:05.417803 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:05.417758 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 23:44:05.418296 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:05.417826 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 23:44:05.583070 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:05.581335 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:05.730311 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:44:05.730274 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:44:05.774651 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:05.774519 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:05.844140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:05.844108 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 23:44:05.844361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:05.844162 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 23:44:06.416301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:06.416264 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 23:44:06.416481 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:06.416319 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 23:44:06.843801 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:06.843753 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 23:44:06.844222 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:06.843820 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 23:44:07.220576 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.220539 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:07.416151 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.416115 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 23:44:07.416333 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.416173 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 23:44:07.818299 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.818263 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z"] Nov 03 23:44:07.822528 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.822504 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.834756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.833565 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.834756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.833602 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.834756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.833644 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.834756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.833824 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6h9l\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-kube-api-access-g6h9l\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.834756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.833902 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.834756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.834009 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.834756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.834106 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.834756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.834213 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.834756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.834249 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.836747 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.836717 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z"] Nov 03 23:44:07.843578 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.843548 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 23:44:07.843725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.843602 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 23:44:07.935139 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935107 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6h9l\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-kube-api-access-g6h9l\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935165 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935208 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935241 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935267 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935287 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935339 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935355 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935382 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935846 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935656 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-credential-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935846 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935755 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-data\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.935952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.935905 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-certs\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.936047 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.936015 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-socket\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.936589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.936560 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istiod-ca-cert\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.937618 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.937597 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-envoy\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.938017 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.938002 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-podinfo\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.947927 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.947903 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-token\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:07.949196 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:07.949167 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6h9l\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-kube-api-access-g6h9l\") pod \"e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:08.138778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:08.138670 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:08.392968 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:08.392883 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:08.415866 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:08.415824 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 23:44:08.416052 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:08.415894 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 23:44:08.514933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:08.492126 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z"] Nov 03 23:44:08.524775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:08.524669 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:08.525048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:08.525002 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:08.525225 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:08.525204 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:09.252444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:09.252404 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" event={"ID":"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf","Type":"ContainerStarted","Data":"289230f4ef2e10a7a06419fa4eb552f73dd70598199c41dc4e8f36475c2ea091"} Nov 03 23:44:09.252877 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:09.252476 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" event={"ID":"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf","Type":"ContainerStarted","Data":"5d25f0040b8f71ac501694508fdb27314d1b972b82b66a06a0c20b4fc970a55a"} Nov 03 23:44:09.332360 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:09.327447 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" podStartSLOduration=2.327426679 podStartE2EDuration="2.327426679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:09.313592057 +0000 UTC m=+868.136188219" watchObservedRunningTime="2026-11-03 23:44:09.327426679 +0000 UTC m=+868.150022841" Nov 03 23:44:09.416345 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:09.416307 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 23:44:09.416500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:09.416360 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 23:44:09.613219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:09.613121 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:09.893299 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:09.893160 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:44:09.893561 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:09.893510 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:10.142417 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.142382 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:10.144128 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.144026 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 23:44:10.144326 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.144299 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 23:44:10.420871 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.416411 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 23:44:10.420871 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.416472 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 23:44:10.539003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.538954 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:10.539549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.539520 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:10.540364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.540340 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:44:10.740484 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.740453 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:10.752851 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:10.752821 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:11.139931 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:11.139828 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 23:44:11.139931 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:11.139888 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 23:44:11.240724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:11.240688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:11.326502 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:11.326442 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:11.739240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:11.739208 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:11.740241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:11.739803 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:11.741430 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:11.741398 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 03 23:44:12.139749 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:12.139665 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 23:44:12.139749 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:12.139724 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 23:44:12.573932 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:12.573856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:13.139898 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:13.139857 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 23:44:13.140455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:13.139932 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 23:44:13.411332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:13.411243 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:13.578063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:13.577930 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:14.139078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:14.139015 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 23:44:14.139265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:14.139103 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 23:44:14.594743 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:14.594699 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:15.140020 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:15.139981 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 23:44:15.140217 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:15.140047 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 23:44:15.552962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:15.552924 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:16.144594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:16.144558 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:16.145027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:16.144942 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:16.145589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:16.145572 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:44:16.385123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:16.385090 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:17.277599 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:17.277562 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:17.546986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:17.546899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:18.363418 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:18.363389 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:19.382436 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:19.382399 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:20.266406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:20.266366 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:21.504343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:21.504300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:23.000220 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:22.999490 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:23.014862 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:23.014789 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:23.317591 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:23.317501 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:23.458114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:23.457143 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:24.106601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:24.106561 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:24.864135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:24.864089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:24.983514 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:24.983479 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:25.091555 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:25.091509 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:25.867500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:25.867439 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:26.983743 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:26.983700 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:27.884200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:27.884163 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:28.317484 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:28.317450 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:29.364270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:29.364233 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:30.287869 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:30.287833 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:31.229607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:31.229573 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:32.178294 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:32.178258 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:33.270779 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:33.270736 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:34.149253 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:34.149211 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:35.050970 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:35.050928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:35.704962 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:44:35.704915 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:44:35.944518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:35.944480 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:36.757493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:36.757450 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:37.640691 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:37.640658 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:37.796850 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:37.796813 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:37.931572 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:37.931485 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:38.657547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:38.657511 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:39.947684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:39.947648 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:41.007057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:41.006150 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:41.052058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:41.045140 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:41.203059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:41.196141 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:41.246845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:41.246764 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:41.325443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:41.325351 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 23:44:41.460242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:41.460199 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:41.768089 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:41.759106 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:41.865438 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:41.865408 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:44:41.877303 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:41.877271 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:44:42.008133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.006386 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 23:44:42.008133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.007019 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:44:42.045618 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.045514 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:42.046749 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.046722 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:44:42.200190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.200146 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 23:44:42.210065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.201265 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:44:42.317120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.273990 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 23:44:42.317120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.274973 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:44:42.426093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.340779 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 23:44:42.428836 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.427931 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:42.473099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.466602 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:42.473099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.467017 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:44:42.519861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.519584 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:42.967528 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.967496 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:44:42.973457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:42.973417 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:43.038458 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.037895 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:43.074267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.074220 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 23:44:43.249463 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.205297 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:43.310419 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.305178 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 23:44:43.359460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.327129 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:43.460533 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.453407 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:43.492419 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.467656 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:44:43.627743 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.627653 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:44:43.628342 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.628322 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:44:43.694068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.694016 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:44:43.923504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.923422 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:44:43.926638 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:43.926614 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:44:44.045064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:44.043574 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:44:44.650948 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:44.650914 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:45.437154 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:45.437114 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:46.267892 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:46.267858 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:47.210271 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.210227 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:47.612511 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.612416 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm"] Nov 03 23:44:47.618488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.618462 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.643329 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.643298 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm"] Nov 03 23:44:47.786864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.786830 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2385f4-0efd-4c2b-8291-327293f0db79-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.786864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.786871 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.787137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.787064 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.787137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.787111 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.787279 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.787261 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnzdz\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-kube-api-access-vnzdz\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.787394 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.787370 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.787467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.787425 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.787530 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.787517 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.787587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.787544 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.888987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.888892 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.888987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.888941 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889010 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnzdz\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-kube-api-access-vnzdz\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889083 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889111 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889130 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889146 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889242 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2385f4-0efd-4c2b-8291-327293f0db79-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889272 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889613 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889589 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889619 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-data\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889757 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-certs\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.889893 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889872 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-credential-socket\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.890004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.889984 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2385f4-0efd-4c2b-8291-327293f0db79-istiod-ca-cert\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.891526 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.891503 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-envoy\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.891683 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.891666 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-podinfo\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.901464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.901441 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-token\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.901591 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.901570 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnzdz\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-kube-api-access-vnzdz\") pod \"e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.934311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.934286 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:47.944045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:47.944004 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:48.059905 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:48.059871 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:48.234976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:48.234946 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm"] Nov 03 23:44:48.235867 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:44:48.235837 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2385f4_0efd_4c2b_8291_327293f0db79.slice/crio-c0c257c45959f9a880139cc8934963728590e23a0a2b5b9f8a36c813cd143b25 WatchSource:0}: Error finding container c0c257c45959f9a880139cc8934963728590e23a0a2b5b9f8a36c813cd143b25: Status 404 returned error can't find the container with id c0c257c45959f9a880139cc8934963728590e23a0a2b5b9f8a36c813cd143b25 Nov 03 23:44:48.238476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:48.238441 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:48.238566 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:48.238506 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:48.238566 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:48.238535 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:44:48.995654 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:48.995621 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:49.087878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:49.087810 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" event={"ID":"7a2385f4-0efd-4c2b-8291-327293f0db79","Type":"ContainerStarted","Data":"9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669"} Nov 03 23:44:49.087878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:49.087881 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" event={"ID":"7a2385f4-0efd-4c2b-8291-327293f0db79","Type":"ContainerStarted","Data":"c0c257c45959f9a880139cc8934963728590e23a0a2b5b9f8a36c813cd143b25"} Nov 03 23:44:49.314052 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:49.313935 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:49.425811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:49.425774 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:49.935175 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:49.935141 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:49.936788 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:49.936765 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 23:44:49.936910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:49.936804 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 23:44:50.408879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:50.408839 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:50.935595 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:50.935559 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 23:44:50.935781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:50.935626 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 23:44:51.590301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:51.590268 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:51.935243 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:51.935150 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 23:44:51.935243 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:51.935222 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 23:44:52.910782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:52.910743 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:52.934779 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:52.934741 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 23:44:52.934956 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:52.934798 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 23:44:53.801460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:53.801315 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:53.936452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:53.936410 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 23:44:53.936968 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:53.936464 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 23:44:54.810540 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:54.810511 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:54.934901 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:54.934869 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 23:44:54.935123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:54.934919 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 23:44:55.580727 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:55.580685 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:55.838872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:55.838780 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:55.941269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:55.941239 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:55.941713 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:55.941690 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:55.942537 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:55.942516 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 03 23:44:55.970875 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:55.970824 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" podStartSLOduration=8.970810261 podStartE2EDuration="8.970810261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:44:49.177479284 +0000 UTC m=+908.000075447" watchObservedRunningTime="2026-11-03 23:44:55.970810261 +0000 UTC m=+914.793406423" Nov 03 23:44:56.670442 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:56.670411 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:57.445909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:57.445875 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:58.346813 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:58.346784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:59.247055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.243813 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:44:59.329364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.329331 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9"] Nov 03 23:44:59.334644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.334614 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9"] Nov 03 23:44:59.334785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.334738 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.402970 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.402925 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.403431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.402990 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.403431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.403081 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.403431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.403132 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.403431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.403153 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.403431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.403240 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zknc9\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-kube-api-access-zknc9\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.403431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.403263 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.403431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.403366 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.403685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.403468 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.512109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.511197 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.512109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.511259 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.512109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.511288 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.512109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.511313 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.512109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.511355 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.512109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.511426 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknc9\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-kube-api-access-zknc9\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.512109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.511456 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.512109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.511547 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.512109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.511642 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.513483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.513454 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.535289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.514494 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.535514 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.514708 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.535617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.514917 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.535709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.534882 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.535809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.535211 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.539202 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.539175 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.559617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.559584 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.561789 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.561755 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zknc9\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-kube-api-access-zknc9\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:44:59.664058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:44:59.663351 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:45:00.595910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.533462 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz"] Nov 03 23:45:00.595910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.562694 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:00.595910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.570477 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4135d67-ead2-4c81-8506-c04854d7c737-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz\" (UID: \"c4135d67-ead2-4c81-8506-c04854d7c737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:00.595910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.570526 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldcsq\" (UniqueName: \"kubernetes.io/projected/c4135d67-ead2-4c81-8506-c04854d7c737-kube-api-access-ldcsq\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz\" (UID: \"c4135d67-ead2-4c81-8506-c04854d7c737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:00.595910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.593403 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\"" Nov 03 23:45:00.622575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.622483 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz"] Nov 03 23:45:00.639264 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.639151 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9"] Nov 03 23:45:00.776767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.749169 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4135d67-ead2-4c81-8506-c04854d7c737-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz\" (UID: \"c4135d67-ead2-4c81-8506-c04854d7c737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:00.776767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.749222 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldcsq\" (UniqueName: \"kubernetes.io/projected/c4135d67-ead2-4c81-8506-c04854d7c737-kube-api-access-ldcsq\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz\" (UID: \"c4135d67-ead2-4c81-8506-c04854d7c737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:00.776767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.750488 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4135d67-ead2-4c81-8506-c04854d7c737-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz\" (UID: \"c4135d67-ead2-4c81-8506-c04854d7c737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:00.781066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.780433 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:00.781066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.780538 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:00.781066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.780585 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:00.924517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.918819 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp"] Nov 03 23:45:00.957438 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.957335 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:00.991108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.976945 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.002122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:00.993144 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldcsq\" (UniqueName: \"kubernetes.io/projected/c4135d67-ead2-4c81-8506-c04854d7c737-kube-api-access-ldcsq\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz\" (UID: \"c4135d67-ead2-4c81-8506-c04854d7c737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:01.141396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.114492 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.141396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.114558 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.141396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.114645 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.141396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.114694 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.141396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.114762 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.141396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.114940 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.141396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.114966 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bce0247-4a53-4c8b-9c84-152d2b268905-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.141396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.114998 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4wgq\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-kube-api-access-g4wgq\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.141396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.115058 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.168139 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.142848 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.291154 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp"] Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.293338 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.315089 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.330333 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.333453 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.333510 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.333570 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.333694 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.333721 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bce0247-4a53-4c8b-9c84-152d2b268905-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.333755 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wgq\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-kube-api-access-g4wgq\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.333783 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.333868 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.334650 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.333909 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.352116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.347204 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bce0247-4a53-4c8b-9c84-152d2b268905-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.353653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.352830 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.353653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.353121 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.353653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.353365 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.353653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.353600 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.433059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.380775 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:01.480466 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.480136 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wgq\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-kube-api-access-g4wgq\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.515369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.480604 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.515369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.492152 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:01.559013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.558956 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" event={"ID":"d633bda6-7b36-4e87-b1fe-e896c5c9072c","Type":"ContainerStarted","Data":"0db13e03138001200fb539ee0f97eca16980ab2bec3450b7713f90cd57bb5275"} Nov 03 23:45:01.595363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.589967 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:01.615062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.608561 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.637455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.637420 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.689205 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.687325 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:01.767200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.737327 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:01.767200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:01.762143 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:02.358065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.357264 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 23:45:02.368074 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.358282 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:45:02.382065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.373684 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:02.402064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.382967 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:45:02.441058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.402979 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:02.441865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.441836 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:45:02.509203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.509146 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:02.509591 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.509559 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 23:45:02.518063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.510192 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:45:02.596222 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.596181 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz"] Nov 03 23:45:02.706064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.701236 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 23:45:02.723063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.711170 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:45:02.723890 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.723858 6208 patch_prober.go:29] interesting pod/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:45:02.724126 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.724093 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:02.770565 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:02.765443 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:03.107190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.094284 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:45:03.214644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.187231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:03.235821 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.235794 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:45:03.374697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.374650 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:03.392939 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.392892 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:03.456174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.456127 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 23:45:03.518314 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.518262 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:03.708128 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.658227 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" event={"ID":"d633bda6-7b36-4e87-b1fe-e896c5c9072c","Type":"ContainerStarted","Data":"6fbc68a0b5851a14ce8caf789c4485e922742fb2104eba9fefad65f0d1c0bb47"} Nov 03 23:45:03.986120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.983709 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:45:03.986120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.826505 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:03.991519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.780748 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 23:45:03.997635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.845318 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 23:45:03.997883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.997854 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 23:45:04.001145 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:03.850161 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podStartSLOduration=4.850139707 podStartE2EDuration="4.850139707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:03.846763168 +0000 UTC m=+922.669359333" watchObservedRunningTime="2026-11-03 23:45:03.850139707 +0000 UTC m=+922.672735870" Nov 03 23:45:04.376467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:04.376414 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:04.396153 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:04.396109 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:04.486501 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:04.486451 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:04.538227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:04.538175 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 23:45:04.718061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:04.710093 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:45:04.718061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:04.715273 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 23:45:04.718061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:04.715326 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 23:45:04.752802 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:04.752767 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:45:05.021996 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.007331 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:05.103141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.102419 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" event={"ID":"c4135d67-ead2-4c81-8506-c04854d7c737","Type":"ContainerStarted","Data":"e094200172d7e6d9e762f3071c2db1981eab13ea69c6cc57935e2281870fd964"} Nov 03 23:45:05.231423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.231372 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:05.231717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.231690 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:05.231879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.231857 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:05.393106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.378496 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:05.536222 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.536122 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:05.566061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.565138 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:05.707057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.681682 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 23:45:05.707057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.681740 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 23:45:05.731059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.712125 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:05.823135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.822547 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp"] Nov 03 23:45:05.866578 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.866542 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:05.894140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:05.894095 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz"] Nov 03 23:45:06.008930 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.008884 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:06.034267 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:45:06.033142 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:45:06.141790 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.141706 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:06.146466 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.146437 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.189062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.166146 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:45:06.189062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.166518 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:45:06.189062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.166592 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:45:06.207371 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.201799 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz"] Nov 03 23:45:06.215355 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.215325 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:45:06.246180 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.246144 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" event={"ID":"7bce0247-4a53-4c8b-9c84-152d2b268905","Type":"ContainerStarted","Data":"ca8af744dd6a77d78439c7162a86faa4506ceaf8d0c66d1f9c6181967e8bee0b"} Nov 03 23:45:06.277852 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.277814 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:45:06.326538 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.324970 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a5ff3c8-85vxz\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.326538 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.325051 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a5ff3c8-85vxz\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.326538 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.325231 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7nw\" (UniqueName: \"kubernetes.io/projected/071f7045-14b1-45ae-a81e-ab4de2ee93c5-kube-api-access-st7nw\") pod \"maas-api-revoke-sub-7a5ff3c8-85vxz\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.426446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.426354 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7nw\" (UniqueName: \"kubernetes.io/projected/071f7045-14b1-45ae-a81e-ab4de2ee93c5-kube-api-access-st7nw\") pod \"maas-api-revoke-sub-7a5ff3c8-85vxz\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.426649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.426626 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a5ff3c8-85vxz\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.426724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.426672 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a5ff3c8-85vxz\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.427723 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.427408 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a5ff3c8-85vxz\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.431076 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.431052 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a5ff3c8-85vxz\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.456320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.456236 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7nw\" (UniqueName: \"kubernetes.io/projected/071f7045-14b1-45ae-a81e-ab4de2ee93c5-kube-api-access-st7nw\") pod \"maas-api-revoke-sub-7a5ff3c8-85vxz\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.612097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.611803 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:06.690109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.689365 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 23:45:06.690109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.689429 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 23:45:06.991447 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:06.990309 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz"] Nov 03 23:45:07.000450 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:45:07.000423 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071f7045_14b1_45ae_a81e_ab4de2ee93c5.slice/crio-41ceeb8849ca488c5c448c6b74fdacbf2d67c6e5bc17e46ffc8979024dd4be01 WatchSource:0}: Error finding container 41ceeb8849ca488c5c448c6b74fdacbf2d67c6e5bc17e46ffc8979024dd4be01: Status 404 returned error can't find the container with id 41ceeb8849ca488c5c448c6b74fdacbf2d67c6e5bc17e46ffc8979024dd4be01 Nov 03 23:45:07.209059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:07.199870 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:07.345141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:07.343750 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" event={"ID":"071f7045-14b1-45ae-a81e-ab4de2ee93c5","Type":"ContainerStarted","Data":"41ceeb8849ca488c5c448c6b74fdacbf2d67c6e5bc17e46ffc8979024dd4be01"} Nov 03 23:45:07.345141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:07.343801 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" event={"ID":"7bce0247-4a53-4c8b-9c84-152d2b268905","Type":"ContainerStarted","Data":"9688b0138abe642150a9c15c017e6400e40f18627f37dad785c4f97079e4ace7"} Nov 03 23:45:07.422223 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:07.421861 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" podStartSLOduration=7.421839991 podStartE2EDuration="7.421839991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:07.398485358 +0000 UTC m=+926.221081519" watchObservedRunningTime="2026-11-03 23:45:07.421839991 +0000 UTC m=+926.244436154" Nov 03 23:45:07.706123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:07.702096 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:07.713085 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:07.712804 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 23:45:07.713085 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:07.712867 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 23:45:07.713302 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:07.713188 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 23:45:07.713302 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:07.713222 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 23:45:08.308436 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:08.308405 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:08.684109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:08.681154 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 23:45:08.684109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:08.681216 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 23:45:08.689553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:08.688362 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 23:45:08.689553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:08.688417 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 23:45:09.014100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:09.013904 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:09.709091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:09.706677 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 23:45:09.709091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:09.706727 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 23:45:09.709091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:09.707054 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 23:45:09.709091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:09.707081 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 23:45:09.709091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:09.707692 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:10.674685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:10.674228 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 23:45:10.674685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:10.674290 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 23:45:10.688242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:10.688176 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 23:45:10.688242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:10.688234 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 23:45:10.791615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:10.791574 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:11.404242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.404195 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" event={"ID":"071f7045-14b1-45ae-a81e-ab4de2ee93c5","Type":"ContainerStarted","Data":"4dc03062f3246e8527da0516b18419b6bc9b51c08186d28b48fea6b0f25e4d50"} Nov 03 23:45:11.406905 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.406867 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" event={"ID":"c4135d67-ead2-4c81-8506-c04854d7c737","Type":"ContainerStarted","Data":"b5027793b09c031a689f68b56d2b3828f3f3f30d8c4b830680e2d0cb6cc16cdf"} Nov 03 23:45:11.429944 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.429828 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" podStartSLOduration=2.695047448 podStartE2EDuration="6.429809159s" totalImagesPullingTime="3.734761711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:11.423739714 +0000 UTC m=+930.246335883" watchObservedRunningTime="2026-11-03 23:45:11.429809159 +0000 UTC m=+930.252405322" Nov 03 23:45:11.443198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.442958 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" podStartSLOduration=3.991753852 podStartE2EDuration="11.44294457s" totalImagesPullingTime="7.451190718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:11.442528296 +0000 UTC m=+930.265124459" watchObservedRunningTime="2026-11-03 23:45:11.44294457 +0000 UTC m=+930.265540726" Nov 03 23:45:11.688401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.688314 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:45:11.688574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.688440 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 23:45:11.688574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.688479 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 23:45:11.689278 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.689194 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:45:11.690561 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.690538 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 03 23:45:11.802946 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:11.802910 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:12.415718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:12.415637 6208 generic.go:356] "Generic (PLEG): container finished" podID="c4135d67-ead2-4c81-8506-c04854d7c737" containerID="b5027793b09c031a689f68b56d2b3828f3f3f30d8c4b830680e2d0cb6cc16cdf" exitCode=60 Nov 03 23:45:12.415881 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:12.415721 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" event={"ID":"c4135d67-ead2-4c81-8506-c04854d7c737","Type":"ContainerDied","Data":"b5027793b09c031a689f68b56d2b3828f3f3f30d8c4b830680e2d0cb6cc16cdf"} Nov 03 23:45:12.416014 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:12.416000 6208 scope.go:122] "RemoveContainer" containerID="b5027793b09c031a689f68b56d2b3828f3f3f30d8c4b830680e2d0cb6cc16cdf" Nov 03 23:45:12.417445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:12.417421 6208 generic.go:356] "Generic (PLEG): container finished" podID="071f7045-14b1-45ae-a81e-ab4de2ee93c5" containerID="4dc03062f3246e8527da0516b18419b6bc9b51c08186d28b48fea6b0f25e4d50" exitCode=0 Nov 03 23:45:12.417571 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:12.417527 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" event={"ID":"071f7045-14b1-45ae-a81e-ab4de2ee93c5","Type":"ContainerDied","Data":"4dc03062f3246e8527da0516b18419b6bc9b51c08186d28b48fea6b0f25e4d50"} Nov 03 23:45:12.688832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:12.688751 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 23:45:12.688832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:12.688811 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 23:45:12.850364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:12.850334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:13.427580 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.427540 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" event={"ID":"c4135d67-ead2-4c81-8506-c04854d7c737","Type":"ContainerStarted","Data":"3158c6635f118df5fe77648c6febcfa06c1fc5c4205c842388a04495059b9515"} Nov 03 23:45:13.680687 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.680614 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:13.695125 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.695087 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-xks-ca\") pod \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " Nov 03 23:45:13.695303 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.695142 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-openshift-ca\") pod \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " Nov 03 23:45:13.695384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.695355 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/071f7045-14b1-45ae-a81e-ab4de2ee93c5-kube-api-access-st7nw\" (UniqueName: \"kubernetes.io/projected/071f7045-14b1-45ae-a81e-ab4de2ee93c5-kube-api-access-st7nw\") pod \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\" (UID: \"071f7045-14b1-45ae-a81e-ab4de2ee93c5\") " Nov 03 23:45:13.695604 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.695571 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-openshift-ca" pod "071f7045-14b1-45ae-a81e-ab4de2ee93c5" (UID: "071f7045-14b1-45ae-a81e-ab4de2ee93c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:45:13.695995 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.695961 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:45:13.697429 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.697396 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071f7045-14b1-45ae-a81e-ab4de2ee93c5-kube-api-access-st7nw" pod "071f7045-14b1-45ae-a81e-ab4de2ee93c5" (UID: "071f7045-14b1-45ae-a81e-ab4de2ee93c5"). InnerVolumeSpecName "kube-api-access-st7nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:45:13.697592 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.697566 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-xks-ca" pod "071f7045-14b1-45ae-a81e-ab4de2ee93c5" (UID: "071f7045-14b1-45ae-a81e-ab4de2ee93c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:45:13.797970 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.797919 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7nw\" (UniqueName: \"kubernetes.io/projected/071f7045-14b1-45ae-a81e-ab4de2ee93c5-kube-api-access-st7nw\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:45:13.797970 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.797964 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/071f7045-14b1-45ae-a81e-ab4de2ee93c5-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:45:13.834075 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:13.832935 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:14.197506 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.197473 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:14.198096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.198019 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:14.198891 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.198875 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:45:14.435852 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.435801 6208 generic.go:356] "Generic (PLEG): container finished" podID="c4135d67-ead2-4c81-8506-c04854d7c737" containerID="3158c6635f118df5fe77648c6febcfa06c1fc5c4205c842388a04495059b9515" exitCode=60 Nov 03 23:45:14.436271 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.435909 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" event={"ID":"c4135d67-ead2-4c81-8506-c04854d7c737","Type":"ContainerDied","Data":"3158c6635f118df5fe77648c6febcfa06c1fc5c4205c842388a04495059b9515"} Nov 03 23:45:14.436271 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.435963 6208 scope.go:122] "RemoveContainer" containerID="b5027793b09c031a689f68b56d2b3828f3f3f30d8c4b830680e2d0cb6cc16cdf" Nov 03 23:45:14.436423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.436301 6208 scope.go:122] "RemoveContainer" containerID="3158c6635f118df5fe77648c6febcfa06c1fc5c4205c842388a04495059b9515" Nov 03 23:45:14.436737 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:45:14.436665 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz_odh-ai-gateway-infra(c4135d67-ead2-4c81-8506-c04854d7c737)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" Nov 03 23:45:14.438143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.438111 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" event={"ID":"071f7045-14b1-45ae-a81e-ab4de2ee93c5","Type":"ContainerDied","Data":"41ceeb8849ca488c5c448c6b74fdacbf2d67c6e5bc17e46ffc8979024dd4be01"} Nov 03 23:45:14.438225 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.438149 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz" Nov 03 23:45:14.438225 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.438159 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41ceeb8849ca488c5c448c6b74fdacbf2d67c6e5bc17e46ffc8979024dd4be01" Nov 03 23:45:14.656456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:14.656417 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:15.455312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:15.455282 6208 scope.go:122] "RemoveContainer" containerID="3158c6635f118df5fe77648c6febcfa06c1fc5c4205c842388a04495059b9515" Nov 03 23:45:15.455727 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:45:15.455515 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz_odh-ai-gateway-infra(c4135d67-ead2-4c81-8506-c04854d7c737)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" Nov 03 23:45:15.625528 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:15.625489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:16.366625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:16.366593 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:17.080778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.080737 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:17.369607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.369514 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:17.525438 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.525400 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:17.541200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.541166 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm"] Nov 03 23:45:17.542088 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.542062 6208 cpu_manager.go:414] "containerMap: removing container" podUID="071f7045-14b1-45ae-a81e-ab4de2ee93c5" containerName="revoke-keys" Nov 03 23:45:17.542088 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.542085 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071f7045-14b1-45ae-a81e-ab4de2ee93c5" containerName="revoke-keys" Nov 03 23:45:17.542311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.542298 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="071f7045-14b1-45ae-a81e-ab4de2ee93c5" containerName="revoke-keys" Nov 03 23:45:17.549992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.547966 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.552220 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.552189 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm"] Nov 03 23:45:17.585684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.583281 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-88f34dbe-openshift-default-dockercfg-nhpxr\"" Nov 03 23:45:17.592259 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.592232 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9"] Nov 03 23:45:17.597248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.597224 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.616122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.616098 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9"] Nov 03 23:45:17.638694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.638615 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:17.654807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.654771 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.655017 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.654825 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-envoy\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.655017 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.654874 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-token\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.655017 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.654930 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-data\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.655017 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.655009 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-podinfo\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.655370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.655081 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptj2r\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-kube-api-access-ptj2r\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.655370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.655146 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e009c7e2-89eb-42bc-bbfb-24decb053b12-istiod-ca-cert\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.655370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.655179 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-certs\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.655370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.655219 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w626q\" (UniqueName: \"kubernetes.io/projected/e2e70338-2c45-448d-bacd-e6e8ff5409c7-kube-api-access-w626q\") pod \"maas-api-revoke-sub-24b4fa05-g5hs9\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.655370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.655268 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b4fa05-g5hs9\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.655370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.655326 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b4fa05-g5hs9\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.655677 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.655399 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-credential-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.657168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.657138 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757491 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-token\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757567 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-data\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757644 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-podinfo\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757672 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptj2r\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-kube-api-access-ptj2r\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757724 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e009c7e2-89eb-42bc-bbfb-24decb053b12-istiod-ca-cert\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757754 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-certs\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757790 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w626q\" (UniqueName: \"kubernetes.io/projected/e2e70338-2c45-448d-bacd-e6e8ff5409c7-kube-api-access-w626q\") pod \"maas-api-revoke-sub-24b4fa05-g5hs9\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757835 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b4fa05-g5hs9\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757888 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b4fa05-g5hs9\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.758067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.757957 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-credential-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.758716 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.758100 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.758716 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.758141 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-envoy\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.773349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.758884 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-certs\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.773349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.759635 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-data\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.773349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.770428 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-credential-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.773349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.770691 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-socket\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.773349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.771692 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24b4fa05-g5hs9\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.773349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.772390 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e009c7e2-89eb-42bc-bbfb-24decb053b12-istiod-ca-cert\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.781756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.781723 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-podinfo\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.781973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.781943 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-envoy\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.783718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.783690 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptj2r\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-kube-api-access-ptj2r\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.794794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.794228 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24b4fa05-g5hs9\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.805024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.804961 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-token\") pod \"e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.812026 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.811998 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w626q\" (UniqueName: \"kubernetes.io/projected/e2e70338-2c45-448d-bacd-e6e8ff5409c7-kube-api-access-w626q\") pod \"maas-api-revoke-sub-24b4fa05-g5hs9\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:17.867655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.867616 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:17.893092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.892992 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:17.919399 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:17.919369 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:18.185994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.185962 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm"] Nov 03 23:45:18.186762 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:45:18.186738 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode009c7e2_89eb_42bc_bbfb_24decb053b12.slice/crio-49b8f193d7fdca80142edc9bc4d7507c3f7754bf7c5d794a5f12ae4b4d9a07d6 WatchSource:0}: Error finding container 49b8f193d7fdca80142edc9bc4d7507c3f7754bf7c5d794a5f12ae4b4d9a07d6: Status 404 returned error can't find the container with id 49b8f193d7fdca80142edc9bc4d7507c3f7754bf7c5d794a5f12ae4b4d9a07d6 Nov 03 23:45:18.189137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.189106 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:18.189209 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.189169 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:18.189209 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.189196 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:18.239459 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.239430 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9"] Nov 03 23:45:18.248326 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:45:18.248301 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e70338_2c45_448d_bacd_e6e8ff5409c7.slice/crio-e3b6bc7ec68298967b07c5004a5272c87426f9dea8e2517411b32fc07ff498a5 WatchSource:0}: Error finding container e3b6bc7ec68298967b07c5004a5272c87426f9dea8e2517411b32fc07ff498a5: Status 404 returned error can't find the container with id e3b6bc7ec68298967b07c5004a5272c87426f9dea8e2517411b32fc07ff498a5 Nov 03 23:45:18.340089 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.340049 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:18.483670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.483623 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" event={"ID":"e2e70338-2c45-448d-bacd-e6e8ff5409c7","Type":"ContainerStarted","Data":"e3b6bc7ec68298967b07c5004a5272c87426f9dea8e2517411b32fc07ff498a5"} Nov 03 23:45:18.486965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.486927 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" event={"ID":"e009c7e2-89eb-42bc-bbfb-24decb053b12","Type":"ContainerStarted","Data":"0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69"} Nov 03 23:45:18.487175 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.486985 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" event={"ID":"e009c7e2-89eb-42bc-bbfb-24decb053b12","Type":"ContainerStarted","Data":"49b8f193d7fdca80142edc9bc4d7507c3f7754bf7c5d794a5f12ae4b4d9a07d6"} Nov 03 23:45:18.517538 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.517482 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" podStartSLOduration=1.517467498 podStartE2EDuration="1.517467498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:18.515219501 +0000 UTC m=+937.337815664" watchObservedRunningTime="2026-11-03 23:45:18.517467498 +0000 UTC m=+937.340063717" Nov 03 23:45:18.893310 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:18.893225 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:19.147600 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:19.147502 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:19.248066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:19.248010 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:19.494750 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:19.494715 6208 generic.go:356] "Generic (PLEG): container finished" podID="e2e70338-2c45-448d-bacd-e6e8ff5409c7" containerID="e9fff241b73a561520299b96393cbb430295656c35f272800d4b54d897066359" exitCode=0 Nov 03 23:45:19.494948 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:19.494789 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" event={"ID":"e2e70338-2c45-448d-bacd-e6e8ff5409c7","Type":"ContainerDied","Data":"e9fff241b73a561520299b96393cbb430295656c35f272800d4b54d897066359"} Nov 03 23:45:19.899484 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:19.899409 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:20.388469 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.388440 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:20.502900 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.502866 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:20.503788 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.503768 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:45:20.679934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.679907 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:20.806725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.806694 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-openshift-ca\") pod \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " Nov 03 23:45:20.806885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.806785 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-xks-ca\") pod \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " Nov 03 23:45:20.806885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.806827 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e70338-2c45-448d-bacd-e6e8ff5409c7-kube-api-access-w626q\" (UniqueName: \"kubernetes.io/projected/e2e70338-2c45-448d-bacd-e6e8ff5409c7-kube-api-access-w626q\") pod \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\" (UID: \"e2e70338-2c45-448d-bacd-e6e8ff5409c7\") " Nov 03 23:45:20.807106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.807080 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-openshift-ca" pod "e2e70338-2c45-448d-bacd-e6e8ff5409c7" (UID: "e2e70338-2c45-448d-bacd-e6e8ff5409c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:45:20.807466 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.807438 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:45:20.808843 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.808821 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-xks-ca" pod "e2e70338-2c45-448d-bacd-e6e8ff5409c7" (UID: "e2e70338-2c45-448d-bacd-e6e8ff5409c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:45:20.808941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.808901 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e70338-2c45-448d-bacd-e6e8ff5409c7-kube-api-access-w626q" pod "e2e70338-2c45-448d-bacd-e6e8ff5409c7" (UID: "e2e70338-2c45-448d-bacd-e6e8ff5409c7"). InnerVolumeSpecName "kube-api-access-w626q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:45:20.846277 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.846246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:20.908881 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.908811 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e70338-2c45-448d-bacd-e6e8ff5409c7-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:45:20.908881 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.908837 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w626q\" (UniqueName: \"kubernetes.io/projected/e2e70338-2c45-448d-bacd-e6e8ff5409c7-kube-api-access-w626q\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:45:20.944405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:20.944373 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:21.336080 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:21.336048 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:21.512120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:21.512088 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" Nov 03 23:45:21.512120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:21.512087 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9" event={"ID":"e2e70338-2c45-448d-bacd-e6e8ff5409c7","Type":"ContainerDied","Data":"e3b6bc7ec68298967b07c5004a5272c87426f9dea8e2517411b32fc07ff498a5"} Nov 03 23:45:21.512626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:21.512160 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3b6bc7ec68298967b07c5004a5272c87426f9dea8e2517411b32fc07ff498a5" Nov 03 23:45:22.158112 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.158082 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:22.206372 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.206336 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d"] Nov 03 23:45:22.206915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.206902 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e2e70338-2c45-448d-bacd-e6e8ff5409c7" containerName="revoke-keys" Nov 03 23:45:22.206955 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.206916 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e70338-2c45-448d-bacd-e6e8ff5409c7" containerName="revoke-keys" Nov 03 23:45:22.207063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.207050 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e70338-2c45-448d-bacd-e6e8ff5409c7" containerName="revoke-keys" Nov 03 23:45:22.210402 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.210386 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:22.221203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.221166 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-88f34dbe-dockercfg-h9zmb\"" Nov 03 23:45:22.221203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.221187 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-88f34dbe\"" Nov 03 23:45:22.231782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.231756 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d"] Nov 03 23:45:22.266559 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.266512 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:22.329503 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.329474 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbh6j\" (UniqueName: \"kubernetes.io/projected/e96be928-44af-4ce7-9cae-45d51a370e8f-kube-api-access-dbh6j\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d\" (UID: \"e96be928-44af-4ce7-9cae-45d51a370e8f\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:22.329655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.329594 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96be928-44af-4ce7-9cae-45d51a370e8f-plugins-config-volume\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d\" (UID: \"e96be928-44af-4ce7-9cae-45d51a370e8f\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:22.376653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.376613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:22.444114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.436143 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh6j\" (UniqueName: \"kubernetes.io/projected/e96be928-44af-4ce7-9cae-45d51a370e8f-kube-api-access-dbh6j\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d\" (UID: \"e96be928-44af-4ce7-9cae-45d51a370e8f\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:22.444114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.436265 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96be928-44af-4ce7-9cae-45d51a370e8f-plugins-config-volume\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d\" (UID: \"e96be928-44af-4ce7-9cae-45d51a370e8f\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:22.444114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.437237 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96be928-44af-4ce7-9cae-45d51a370e8f-plugins-config-volume\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d\" (UID: \"e96be928-44af-4ce7-9cae-45d51a370e8f\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:22.463008 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.462411 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbh6j\" (UniqueName: \"kubernetes.io/projected/e96be928-44af-4ce7-9cae-45d51a370e8f-kube-api-access-dbh6j\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d\" (UID: \"e96be928-44af-4ce7-9cae-45d51a370e8f\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:22.525405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.525367 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:22.833007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:22.832979 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d"] Nov 03 23:45:22.833847 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:45:22.833820 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96be928_44af_4ce7_9cae_45d51a370e8f.slice/crio-9530b02f9ab44aaeab63846e4a77a631045692631f3d877a6abd2c27188ff963 WatchSource:0}: Error finding container 9530b02f9ab44aaeab63846e4a77a631045692631f3d877a6abd2c27188ff963: Status 404 returned error can't find the container with id 9530b02f9ab44aaeab63846e4a77a631045692631f3d877a6abd2c27188ff963 Nov 03 23:45:23.260755 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:23.260715 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:23.532121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:23.532008 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" event={"ID":"e96be928-44af-4ce7-9cae-45d51a370e8f","Type":"ContainerStarted","Data":"3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7"} Nov 03 23:45:23.532121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:23.532093 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" event={"ID":"e96be928-44af-4ce7-9cae-45d51a370e8f","Type":"ContainerStarted","Data":"9530b02f9ab44aaeab63846e4a77a631045692631f3d877a6abd2c27188ff963"} Nov 03 23:45:23.532121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:23.532119 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:23.551624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:23.551557 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" podStartSLOduration=1.551543149 podStartE2EDuration="1.551543149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:23.550780669 +0000 UTC m=+942.373376833" watchObservedRunningTime="2026-11-03 23:45:23.551543149 +0000 UTC m=+942.374139311" Nov 03 23:45:24.288992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:24.288941 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:26.021055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:25.981237 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:26.527059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:26.505491 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 23:45:26.613198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:26.613144 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:26.648433 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:26.613464 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:26.779205 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:26.779101 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 23:45:27.014162 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.014118 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:27.137093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.136976 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:27.175054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.171131 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:27.291882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.291842 6208 scope.go:122] "RemoveContainer" containerID="3158c6635f118df5fe77648c6febcfa06c1fc5c4205c842388a04495059b9515" Nov 03 23:45:27.407375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.407276 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:27.508329 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.506816 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:27.508329 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.507557 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:45:27.686121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.683648 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 23:45:27.686121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.684394 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:45:27.812115 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.756161 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:45:27.812115 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:27.756236 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:28.056676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.056105 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:28.066062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.060561 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:45:28.066062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.061172 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:28.088077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.086950 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:45:28.223060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.220735 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 23:45:28.223060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.221932 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:45:28.404909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.363377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:28.517068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.515652 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:28.576056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.563400 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 23:45:28.604058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.603180 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 23:45:28.604058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.603653 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:45:28.604058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.603731 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:45:28.697400 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:28.687205 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:29.057590 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:29.057553 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 23:45:29.101568 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:29.101326 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:29.240447 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:29.240396 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:29.440681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:29.440648 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:45:29.504980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:29.504950 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:45:29.644639 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:29.644549 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Nov 03 23:45:29.653457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:29.653406 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:30.137409 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.137363 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 23:45:30.149931 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.149712 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:45:30.254184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.254134 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:30.375108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.373868 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:45:30.643547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.643469 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:45:30.682919 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.682622 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:30.727056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.718274 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5"] Nov 03 23:45:30.745059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.730290 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:45:30.745059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.730352 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:45:30.757610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.757571 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:30.790712 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.774646 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-88f34dbe-serving-cert\"" Nov 03 23:45:30.899629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.892164 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5"] Nov 03 23:45:30.899629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.896725 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rfs\" (UniqueName: \"kubernetes.io/projected/a02c10da-b946-4464-b57a-4f6246dd0a72-kube-api-access-68rfs\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:30.899629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.896891 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-metrics-certs\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:30.899629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.896947 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-maas-api-tls\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:30.899629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:30.897057 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02c10da-b946-4464-b57a-4f6246dd0a72-openshift-service-ca\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:30.930337 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:45:30.930300 6208 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4135d67_ead2_4c81_8506_c04854d7c737.slice/crio-28ceacd541f2bbbaeeeaf8b7056a8e2387faae6764d73f81f733adf93231484f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 23:45:31.034110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.017500 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02c10da-b946-4464-b57a-4f6246dd0a72-openshift-service-ca\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:31.058560 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.058520 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rfs\" (UniqueName: \"kubernetes.io/projected/a02c10da-b946-4464-b57a-4f6246dd0a72-kube-api-access-68rfs\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:31.058943 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.058918 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-metrics-certs\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:31.059156 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.059135 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-maas-api-tls\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:31.128077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.087731 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02c10da-b946-4464-b57a-4f6246dd0a72-openshift-service-ca\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:31.186542 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.141741 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:31.251137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.217626 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-metrics-certs\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:31.251137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.243111 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-maas-api-tls\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:31.275775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.267916 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rfs\" (UniqueName: \"kubernetes.io/projected/a02c10da-b946-4464-b57a-4f6246dd0a72-kube-api-access-68rfs\") pod \"maas-api-e2e-mt-88f34dbe-884ff66c-p68f5\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:31.275775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.273001 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:45:31.275775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.273922 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:31.278431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.278398 6208 generic.go:356] "Generic (PLEG): container finished" podID="c4135d67-ead2-4c81-8506-c04854d7c737" containerID="28ceacd541f2bbbaeeeaf8b7056a8e2387faae6764d73f81f733adf93231484f" exitCode=60 Nov 03 23:45:31.278692 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.278669 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" event={"ID":"c4135d67-ead2-4c81-8506-c04854d7c737","Type":"ContainerDied","Data":"28ceacd541f2bbbaeeeaf8b7056a8e2387faae6764d73f81f733adf93231484f"} Nov 03 23:45:31.278786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.278737 6208 scope.go:122] "RemoveContainer" containerID="3158c6635f118df5fe77648c6febcfa06c1fc5c4205c842388a04495059b9515" Nov 03 23:45:31.283057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.279244 6208 scope.go:122] "RemoveContainer" containerID="28ceacd541f2bbbaeeeaf8b7056a8e2387faae6764d73f81f733adf93231484f" Nov 03 23:45:31.283057 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:45:31.279606 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz_odh-ai-gateway-infra(c4135d67-ead2-4c81-8506-c04854d7c737)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" Nov 03 23:45:31.353686 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.349371 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:31.744772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:31.742584 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5"] Nov 03 23:45:32.291393 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.291309 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" event={"ID":"a02c10da-b946-4464-b57a-4f6246dd0a72","Type":"ContainerStarted","Data":"05839bc71d3258d30ccf2884605de6cdd4317ada12f7fd0fc2dd11385110e3cb"} Nov 03 23:45:32.385607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.385556 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz"] Nov 03 23:45:32.420921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.420866 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:32.566272 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.566242 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:32.646715 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.646682 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4135d67-ead2-4c81-8506-c04854d7c737-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4135d67-ead2-4c81-8506-c04854d7c737-maas-api-service-ca\") pod \"c4135d67-ead2-4c81-8506-c04854d7c737\" (UID: \"c4135d67-ead2-4c81-8506-c04854d7c737\") " Nov 03 23:45:32.646883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.646790 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4135d67-ead2-4c81-8506-c04854d7c737-kube-api-access-ldcsq\" (UniqueName: \"kubernetes.io/projected/c4135d67-ead2-4c81-8506-c04854d7c737-kube-api-access-ldcsq\") pod \"c4135d67-ead2-4c81-8506-c04854d7c737\" (UID: \"c4135d67-ead2-4c81-8506-c04854d7c737\") " Nov 03 23:45:32.647160 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.647133 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4135d67-ead2-4c81-8506-c04854d7c737-maas-api-service-ca" pod "c4135d67-ead2-4c81-8506-c04854d7c737" (UID: "c4135d67-ead2-4c81-8506-c04854d7c737"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:45:32.647470 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.647453 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4135d67-ead2-4c81-8506-c04854d7c737-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:45:32.649073 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.649007 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4135d67-ead2-4c81-8506-c04854d7c737-kube-api-access-ldcsq" pod "c4135d67-ead2-4c81-8506-c04854d7c737" (UID: "c4135d67-ead2-4c81-8506-c04854d7c737"). InnerVolumeSpecName "kube-api-access-ldcsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:45:32.652544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.652509 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:32.749048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.748997 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldcsq\" (UniqueName: \"kubernetes.io/projected/c4135d67-ead2-4c81-8506-c04854d7c737-kube-api-access-ldcsq\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:45:32.921934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:32.921846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:33.082106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.082025 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:33.387432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.386948 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" event={"ID":"c4135d67-ead2-4c81-8506-c04854d7c737","Type":"ContainerDied","Data":"e094200172d7e6d9e762f3071c2db1981eab13ea69c6cc57935e2281870fd964"} Nov 03 23:45:33.387432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.387027 6208 scope.go:122] "RemoveContainer" containerID="28ceacd541f2bbbaeeeaf8b7056a8e2387faae6764d73f81f733adf93231484f" Nov 03 23:45:33.387432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.387163 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz" Nov 03 23:45:33.397546 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.395617 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" event={"ID":"a02c10da-b946-4464-b57a-4f6246dd0a72","Type":"ContainerStarted","Data":"6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533"} Nov 03 23:45:33.397546 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.396165 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:33.431763 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.431700 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz"] Nov 03 23:45:33.437991 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.437958 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895825-stckz"] Nov 03 23:45:33.473074 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.470233 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" podStartSLOduration=2.94578554 podStartE2EDuration="3.470213532s" totalImagesPullingTime="524.427992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:33.469518491 +0000 UTC m=+952.292114654" watchObservedRunningTime="2026-11-03 23:45:33.470213532 +0000 UTC m=+952.292809697" Nov 03 23:45:33.619521 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:33.619482 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:34.534911 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:34.534871 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:34.605013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:34.604983 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:45:35.296568 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:35.296539 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" path="/var/lib/kubelet/pods/c4135d67-ead2-4c81-8506-c04854d7c737/volumes" Nov 03 23:45:35.366933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:35.366887 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:35.705867 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:45:35.705834 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:45:36.347786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:36.347748 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:37.169045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:37.168990 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:37.284576 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:37.284539 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:37.422506 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:37.422408 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:38.221321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.221286 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:38.258964 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.258928 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg"] Nov 03 23:45:38.259573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.259556 6208 cpu_manager.go:414] "containerMap: removing container" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" containerName="cleanup" Nov 03 23:45:38.259573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.259572 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" containerName="cleanup" Nov 03 23:45:38.259714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.259597 6208 cpu_manager.go:414] "containerMap: removing container" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" containerName="cleanup" Nov 03 23:45:38.259714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.259603 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" containerName="cleanup" Nov 03 23:45:38.259714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.259613 6208 cpu_manager.go:414] "containerMap: removing container" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" containerName="cleanup" Nov 03 23:45:38.259714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.259618 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" containerName="cleanup" Nov 03 23:45:38.259856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.259736 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" containerName="cleanup" Nov 03 23:45:38.259856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.259747 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" containerName="cleanup" Nov 03 23:45:38.259856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.259754 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4135d67-ead2-4c81-8506-c04854d7c737" containerName="cleanup" Nov 03 23:45:38.263946 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.263926 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:38.272606 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.272362 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc"] Nov 03 23:45:38.275627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.275600 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-344261\"" Nov 03 23:45:38.277921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.277900 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg"] Nov 03 23:45:38.278098 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.278024 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:38.291653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.291611 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-344261-dockercfg-v8kgh\"" Nov 03 23:45:38.294259 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.294236 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc"] Nov 03 23:45:38.341872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.341836 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:38.360493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.360458 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568ql\" (UniqueName: \"kubernetes.io/projected/77e0aea2-0333-4c9a-b703-35a9f4bb7473-kube-api-access-568ql\") pod \"payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc\" (UID: \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\") " pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:38.360689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.360591 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e0aea2-0333-4c9a-b703-35a9f4bb7473-plugins-config-volume\") pod \"payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc\" (UID: \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\") " pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:38.360758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.360732 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dfb50d-e44a-49ba-946e-7cd19ef1a513-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg\" (UID: \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:38.360899 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.360879 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xgd9\" (UniqueName: \"kubernetes.io/projected/55dfb50d-e44a-49ba-946e-7cd19ef1a513-kube-api-access-6xgd9\") pod \"payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg\" (UID: \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:38.470062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.460645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:38.484300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.483127 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dfb50d-e44a-49ba-946e-7cd19ef1a513-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg\" (UID: \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:38.484647 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.484627 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xgd9\" (UniqueName: \"kubernetes.io/projected/55dfb50d-e44a-49ba-946e-7cd19ef1a513-kube-api-access-6xgd9\") pod \"payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg\" (UID: \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:38.484916 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.484896 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568ql\" (UniqueName: \"kubernetes.io/projected/77e0aea2-0333-4c9a-b703-35a9f4bb7473-kube-api-access-568ql\") pod \"payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc\" (UID: \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\") " pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:38.485112 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.485094 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e0aea2-0333-4c9a-b703-35a9f4bb7473-plugins-config-volume\") pod \"payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc\" (UID: \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\") " pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:38.504062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.486107 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e0aea2-0333-4c9a-b703-35a9f4bb7473-plugins-config-volume\") pod \"payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc\" (UID: \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\") " pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:38.504337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.484125 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dfb50d-e44a-49ba-946e-7cd19ef1a513-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg\" (UID: \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:38.547999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.547964 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-568ql\" (UniqueName: \"kubernetes.io/projected/77e0aea2-0333-4c9a-b703-35a9f4bb7473-kube-api-access-568ql\") pod \"payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc\" (UID: \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\") " pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:38.549447 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.549271 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xgd9\" (UniqueName: \"kubernetes.io/projected/55dfb50d-e44a-49ba-946e-7cd19ef1a513-kube-api-access-6xgd9\") pod \"payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg\" (UID: \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:38.585479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.585440 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:38.600601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.600573 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:38.900011 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.899978 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg"] Nov 03 23:45:38.901020 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:45:38.900992 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dfb50d_e44a_49ba_946e_7cd19ef1a513.slice/crio-10a143a8c8f1a813022be4c3ac14443c364ea0f3dd9049cb61f00fde407cd57d WatchSource:0}: Error finding container 10a143a8c8f1a813022be4c3ac14443c364ea0f3dd9049cb61f00fde407cd57d: Status 404 returned error can't find the container with id 10a143a8c8f1a813022be4c3ac14443c364ea0f3dd9049cb61f00fde407cd57d Nov 03 23:45:38.941153 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:38.941118 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc"] Nov 03 23:45:38.944313 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:45:38.944264 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e0aea2_0333_4c9a_b703_35a9f4bb7473.slice/crio-15f6d579a90b63b2caf018e9264298cf71ead8c304ea6f3f0175c6412b7d2cbe WatchSource:0}: Error finding container 15f6d579a90b63b2caf018e9264298cf71ead8c304ea6f3f0175c6412b7d2cbe: Status 404 returned error can't find the container with id 15f6d579a90b63b2caf018e9264298cf71ead8c304ea6f3f0175c6412b7d2cbe Nov 03 23:45:39.258268 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.258181 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:39.411412 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.411370 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:45:39.578594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.578497 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" event={"ID":"77e0aea2-0333-4c9a-b703-35a9f4bb7473","Type":"ContainerStarted","Data":"8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6"} Nov 03 23:45:39.578594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.578557 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" event={"ID":"77e0aea2-0333-4c9a-b703-35a9f4bb7473","Type":"ContainerStarted","Data":"15f6d579a90b63b2caf018e9264298cf71ead8c304ea6f3f0175c6412b7d2cbe"} Nov 03 23:45:39.578870 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.578727 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:39.580305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.580282 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" event={"ID":"55dfb50d-e44a-49ba-946e-7cd19ef1a513","Type":"ContainerStarted","Data":"f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33"} Nov 03 23:45:39.580492 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.580318 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" event={"ID":"55dfb50d-e44a-49ba-946e-7cd19ef1a513","Type":"ContainerStarted","Data":"10a143a8c8f1a813022be4c3ac14443c364ea0f3dd9049cb61f00fde407cd57d"} Nov 03 23:45:39.580492 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.580399 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:39.600573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.600521 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" podStartSLOduration=1.600507235 podStartE2EDuration="1.600507235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:39.597118699 +0000 UTC m=+958.419714860" watchObservedRunningTime="2026-11-03 23:45:39.600507235 +0000 UTC m=+958.423103396" Nov 03 23:45:39.619577 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:39.619522 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" podStartSLOduration=1.6195065830000002 podStartE2EDuration="1.619506583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:39.615776315 +0000 UTC m=+958.438372481" watchObservedRunningTime="2026-11-03 23:45:39.619506583 +0000 UTC m=+958.442102744" Nov 03 23:45:40.054858 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:40.054818 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:40.838965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:40.838926 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:41.640275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:41.640236 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:42.419289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:42.419241 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:42.524228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:42.524190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:42.676959 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:42.676873 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:43.399949 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:43.399921 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:44.236308 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:44.236279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:45.111619 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:45.111584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:45.978629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:45.978584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:46.841610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:46.841577 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:47.746446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:47.746413 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:48.139114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:48.139009 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:48.239025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:48.238992 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:48.673799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:48.673740 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:49.358498 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:49.358462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:49.483166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:49.483134 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:49.691109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:49.689149 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:50.626314 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:50.626280 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:45:50.627689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:50.627667 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:45:50.950242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:50.950142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:51.218655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:51.218611 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:51.879902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:51.879865 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:51.990081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:51.986460 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:52.619093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:52.619058 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:52.828622 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:52.828585 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:52.993545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:52.991215 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc"] Nov 03 23:45:52.996388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:52.996349 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:53.003746 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.003235 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc"] Nov 03 23:45:53.003746 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.003321 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.010197 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.010169 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cc7bd0-openshift-default-dockercfg-ssjnp\"" Nov 03 23:45:53.148138 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.148099 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-token\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.148411 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.148383 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-data\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.148643 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.148621 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqz9g\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-kube-api-access-bqz9g\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.148839 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.148823 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3350a74c-ab8f-49db-957c-f32d1dff3241-istiod-ca-cert\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.149055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.149020 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-podinfo\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.149271 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.149253 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-certs\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.149599 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.149580 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-envoy\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.156144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.156118 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-credential-socket\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.156388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.156370 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-socket\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.262931 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-socket\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.263100 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-token\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.263142 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-data\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.263175 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqz9g\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-kube-api-access-bqz9g\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.263229 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3350a74c-ab8f-49db-957c-f32d1dff3241-istiod-ca-cert\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.263299 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-podinfo\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.263360 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-certs\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.263493 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-envoy\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.263540 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-credential-socket\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.264023 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-credential-socket\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.264324 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-socket\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.264846 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-data\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.265670 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3350a74c-ab8f-49db-957c-f32d1dff3241-istiod-ca-cert\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.280113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.268463 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-certs\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.281724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.281653 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-envoy\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.302062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.284014 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-token\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.303070 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.284121 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-podinfo\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.303249 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.302973 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqz9g\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-kube-api-access-bqz9g\") pod \"e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.324772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.324734 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:53.484064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.481444 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:53.685292 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.685264 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc"] Nov 03 23:45:53.692555 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.692516 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:53.692678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.692594 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:53.692678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.692626 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:45:53.846614 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.846577 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:53.915985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.915278 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln"] Nov 03 23:45:53.922433 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.919826 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:53.938123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.936580 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln"] Nov 03 23:45:53.938123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.937605 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-88f34dbe\"/\"openshift-service-ca.crt\"" Nov 03 23:45:53.949172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.949136 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-88f34dbe\"/\"e2e-lifecycle-model-88f34dbe-kserve-self-signed-certs\"" Nov 03 23:45:53.949313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.949178 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-88f34dbe\"/\"kube-root-ca.crt\"" Nov 03 23:45:53.949794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.949769 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-88f34dbe\"/\"default-dockercfg-ldwbc\"" Nov 03 23:45:53.977068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:53.976858 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:54.001680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.000884 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-tmp-dir\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.001680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.000988 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgvn\" (UniqueName: \"kubernetes.io/projected/6de4693f-7d97-4c22-b39c-517473ee9486-kube-api-access-8wgvn\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.001680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.001152 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de4693f-7d97-4c22-b39c-517473ee9486-tls-certs\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.001680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.001228 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-home\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.001680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.001342 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-dshm\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.001680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.001428 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-model-cache\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.024280 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.024239 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" event={"ID":"3350a74c-ab8f-49db-957c-f32d1dff3241","Type":"ContainerStarted","Data":"d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0"} Nov 03 23:45:54.024476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.024303 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" event={"ID":"3350a74c-ab8f-49db-957c-f32d1dff3241","Type":"ContainerStarted","Data":"5a13c32931e02aaec6a5cbfe6e2340ba02e72f9c82c2900ab81ed5fb0c16c58d"} Nov 03 23:45:54.049066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.048968 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" podStartSLOduration=2.04895101 podStartE2EDuration="2.04895101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:54.04740809 +0000 UTC m=+972.870004252" watchObservedRunningTime="2026-11-03 23:45:54.04895101 +0000 UTC m=+972.871547171" Nov 03 23:45:54.103064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.102989 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-model-cache\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.103258 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.103092 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-tmp-dir\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.104717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.103462 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgvn\" (UniqueName: \"kubernetes.io/projected/6de4693f-7d97-4c22-b39c-517473ee9486-kube-api-access-8wgvn\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.104717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.103607 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de4693f-7d97-4c22-b39c-517473ee9486-tls-certs\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.104717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.103683 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-home\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.104717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.103798 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-tmp-dir\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.104717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.103814 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-dshm\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.104717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.104450 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-model-cache\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.105094 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.104725 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-home\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.106454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.106428 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-dshm\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.108194 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.108168 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de4693f-7d97-4c22-b39c-517473ee9486-tls-certs\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.116475 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.116449 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgvn\" (UniqueName: \"kubernetes.io/projected/6de4693f-7d97-4c22-b39c-517473ee9486-kube-api-access-8wgvn\") pod \"e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.273058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.270338 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:54.331143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.331115 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:54.337547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.337512 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:54.703302 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.703269 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln"] Nov 03 23:45:54.876066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:54.875520 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:55.032420 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.032382 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" event={"ID":"6de4693f-7d97-4c22-b39c-517473ee9486","Type":"ContainerStarted","Data":"ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded"} Nov 03 23:45:55.032891 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.032427 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" event={"ID":"6de4693f-7d97-4c22-b39c-517473ee9486","Type":"ContainerStarted","Data":"51eb32883e624572ac9680f4b66aa96bc9e49f52e9e8c9875954bf05da6af5e7"} Nov 03 23:45:55.032891 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.032541 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:55.033613 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.033596 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:45:55.078399 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.078328 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" podStartSLOduration=2.078308549 podStartE2EDuration="2.078308549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:55.052384215 +0000 UTC m=+973.874980377" watchObservedRunningTime="2026-11-03 23:45:55.078308549 +0000 UTC m=+973.900904713" Nov 03 23:45:55.276086 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.273166 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:45:55.643946 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.643916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:55.898342 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.898266 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz"] Nov 03 23:45:55.902602 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.902583 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:55.907826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.907774 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-953960-serving-cert\"" Nov 03 23:45:55.914786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.914762 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz"] Nov 03 23:45:55.988370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:55.988334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:56.048694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.048657 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-metrics-certs\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.049171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.048869 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-maas-api-tls\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.049171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.048915 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a6df0b-8978-4766-a057-f4e8c3993146-openshift-service-ca\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.049171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.049144 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjlvt\" (UniqueName: \"kubernetes.io/projected/21a6df0b-8978-4766-a057-f4e8c3993146-kube-api-access-cjlvt\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.090785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.089164 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:56.150442 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.150366 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-maas-api-tls\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.150442 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.150414 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a6df0b-8978-4766-a057-f4e8c3993146-openshift-service-ca\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.150646 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.150541 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlvt\" (UniqueName: \"kubernetes.io/projected/21a6df0b-8978-4766-a057-f4e8c3993146-kube-api-access-cjlvt\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.151007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.150710 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-metrics-certs\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.151295 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.151269 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a6df0b-8978-4766-a057-f4e8c3993146-openshift-service-ca\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.153067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.153002 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-metrics-certs\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.153187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.153085 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-maas-api-tls\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.163258 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.163232 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlvt\" (UniqueName: \"kubernetes.io/projected/21a6df0b-8978-4766-a057-f4e8c3993146-kube-api-access-cjlvt\") pod \"maas-api-e2e-api-a-953960-567f9c684-5dqlz\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.218932 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.218885 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:56.474976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.474944 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:56.525779 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:45:56.525750 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a6df0b_8978_4766_a057_f4e8c3993146.slice/crio-53bf04fb13669727d2fd030af4b053eea34bc81853c84d44f049b80477a090d4 WatchSource:0}: Error finding container 53bf04fb13669727d2fd030af4b053eea34bc81853c84d44f049b80477a090d4: Status 404 returned error can't find the container with id 53bf04fb13669727d2fd030af4b053eea34bc81853c84d44f049b80477a090d4 Nov 03 23:45:56.526886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.526849 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz"] Nov 03 23:45:56.582981 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:56.582902 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:57.050792 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:57.050754 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" event={"ID":"21a6df0b-8978-4766-a057-f4e8c3993146","Type":"ContainerStarted","Data":"b307704e1064e3d4397f1dffa3d1e261c95b043840a8e70f14cbd1d7b960b587"} Nov 03 23:45:57.051316 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:57.050818 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" event={"ID":"21a6df0b-8978-4766-a057-f4e8c3993146","Type":"ContainerStarted","Data":"53bf04fb13669727d2fd030af4b053eea34bc81853c84d44f049b80477a090d4"} Nov 03 23:45:57.051316 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:57.050980 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:45:57.081768 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:57.081706 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" podStartSLOduration=1.690621652 podStartE2EDuration="2.081688103s" totalImagesPullingTime="391.066451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:45:57.078566559 +0000 UTC m=+975.901162720" watchObservedRunningTime="2026-11-03 23:45:57.081688103 +0000 UTC m=+975.904284268" Nov 03 23:45:57.090552 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:57.090523 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:57.674349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:57.674311 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:57.802760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:57.802717 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:58.149106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:58.145641 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:59.150769 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.150724 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:45:59.472669 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.472621 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7"] Nov 03 23:45:59.495463 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.495429 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7"] Nov 03 23:45:59.495701 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.495477 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.636466 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.636433 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.636634 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.636475 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.636634 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.636580 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.636634 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.636613 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpr8f\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-kube-api-access-qpr8f\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.636743 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.636698 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.636743 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.636722 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.636807 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.636757 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ac3905-17a7-4231-b79a-6cea8e0656b8-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.636878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.636864 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.636913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.636897 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738142 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738064 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738142 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738109 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738142 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738144 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738166 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpr8f\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-kube-api-access-qpr8f\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738212 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738236 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738284 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ac3905-17a7-4231-b79a-6cea8e0656b8-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738354 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738397 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738707 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738632 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738707 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738669 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738819 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738738 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.738904 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.738875 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.739312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.739289 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ac3905-17a7-4231-b79a-6cea8e0656b8-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.740878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.740850 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.740976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.740960 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.750604 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.750581 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpr8f\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-kube-api-access-qpr8f\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.752549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.752525 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.838564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.838529 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:45:59.938882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:45:59.938750 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:00.130211 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:00.130175 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7"] Nov 03 23:46:00.130357 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:46:00.130179 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ac3905_17a7_4231_b79a_6cea8e0656b8.slice/crio-cac591623c7cc10240457126ce5a13a2e887c89a2fe979b0776f268beb14ab0e WatchSource:0}: Error finding container cac591623c7cc10240457126ce5a13a2e887c89a2fe979b0776f268beb14ab0e: Status 404 returned error can't find the container with id cac591623c7cc10240457126ce5a13a2e887c89a2fe979b0776f268beb14ab0e Nov 03 23:46:00.132845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:00.132809 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:00.132926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:00.132876 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:00.132926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:00.132906 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:00.873057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:00.859562 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:01.086867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.086781 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" event={"ID":"33ac3905-17a7-4231-b79a-6cea8e0656b8","Type":"ContainerStarted","Data":"c5e05dcec57c9a69e12754119dc375047bf417d627007ed39882590613af5996"} Nov 03 23:46:01.086867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.086846 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" event={"ID":"33ac3905-17a7-4231-b79a-6cea8e0656b8","Type":"ContainerStarted","Data":"cac591623c7cc10240457126ce5a13a2e887c89a2fe979b0776f268beb14ab0e"} Nov 03 23:46:01.088631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.088607 6208 generic.go:356] "Generic (PLEG): container finished" podID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerID="0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69" exitCode=0 Nov 03 23:46:01.088742 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.088676 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" event={"ID":"e009c7e2-89eb-42bc-bbfb-24decb053b12","Type":"ContainerDied","Data":"0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69"} Nov 03 23:46:01.089070 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.089055 6208 scope.go:122] "RemoveContainer" containerID="0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69" Nov 03 23:46:01.090393 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.090364 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:01.090490 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.090440 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:01.090549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.090492 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:01.114730 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.114685 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" podStartSLOduration=2.114670772 podStartE2EDuration="2.114670772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:01.111941719 +0000 UTC m=+979.934537881" watchObservedRunningTime="2026-11-03 23:46:01.114670772 +0000 UTC m=+979.937267008" Nov 03 23:46:01.838876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.838837 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:46:01.840841 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.840812 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 23:46:01.840967 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.840867 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 23:46:01.865555 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:01.865514 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:02.116666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:02.115474 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" event={"ID":"e009c7e2-89eb-42bc-bbfb-24decb053b12","Type":"ContainerStarted","Data":"28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133"} Nov 03 23:46:02.758397 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:02.758363 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:02.839481 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:02.839448 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 23:46:02.839666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:02.839514 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 23:46:02.901656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:02.901626 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:46:02.913320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:02.913281 6208 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 23:46:02.913608 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:02.913577 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 23:46:03.103417 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:03.103303 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:46:03.698076 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:03.698011 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:03.839564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:03.839529 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 23:46:03.839734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:03.839591 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 23:46:03.893561 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:03.893515 6208 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 23:46:03.893876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:03.893846 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 23:46:04.277058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.276681 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:46:04.315962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.315932 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:46:04.626440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.626313 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq"] Nov 03 23:46:04.632776 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.632754 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq"] Nov 03 23:46:04.632979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.632865 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.637418 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.637345 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:04.638495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.638473 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cc7bd0-serving-cert\"" Nov 03 23:46:04.727376 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.721938 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-maas-api-tls\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.727376 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.722012 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b29n5\" (UniqueName: \"kubernetes.io/projected/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-kube-api-access-b29n5\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.727376 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.722340 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-metrics-certs\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.727376 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.722409 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.736828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.736796 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:04.844066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.841632 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-metrics-certs\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.844066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.841753 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.844066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.841850 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-maas-api-tls\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.844066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.841925 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b29n5\" (UniqueName: \"kubernetes.io/projected/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-kube-api-access-b29n5\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.844066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.843745 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.851003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.844844 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 23:46:04.851003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.844898 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 23:46:04.888103 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.887844 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b29n5\" (UniqueName: \"kubernetes.io/projected/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-kube-api-access-b29n5\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.892056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.890641 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-maas-api-tls\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.896054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.894474 6208 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 23:46:04.896054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.894537 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 23:46:04.902194 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.899612 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-metrics-certs\") pod \"maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:04.950656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:04.950620 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:05.151448 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:05.150267 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:46:05.603845 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:46:05.603793 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad0349e_c7a9_4b96_92e7_7955e6cceeed.slice/crio-e4936d024644e708f413558c71a9a781336032c720f98e97fb0daf69f5416989 WatchSource:0}: Error finding container e4936d024644e708f413558c71a9a781336032c720f98e97fb0daf69f5416989: Status 404 returned error can't find the container with id e4936d024644e708f413558c71a9a781336032c720f98e97fb0daf69f5416989 Nov 03 23:46:05.626648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:05.626597 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:46:05.626809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:05.626662 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq"] Nov 03 23:46:05.640986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:05.640955 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:05.731119 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:46:05.731085 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:46:05.839344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:05.839259 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 23:46:05.839344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:05.839320 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 23:46:05.875059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:05.872312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:05.898540 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:05.898502 6208 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 23:46:05.899692 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:05.899658 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 23:46:06.184465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:06.171704 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" event={"ID":"2ad0349e-c7a9-4b96-92e7-7955e6cceeed","Type":"ContainerStarted","Data":"e4936d024644e708f413558c71a9a781336032c720f98e97fb0daf69f5416989"} Nov 03 23:46:06.848671 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:06.848289 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 23:46:06.848671 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:06.848358 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 23:46:06.894666 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:06.894629 6208 patch_prober.go:29] interesting pod/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 23:46:06.894942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:06.894913 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 23:46:06.931538 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:06.931502 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:07.180079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:07.179968 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" event={"ID":"2ad0349e-c7a9-4b96-92e7-7955e6cceeed","Type":"ContainerStarted","Data":"d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5"} Nov 03 23:46:07.180278 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:07.180257 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:07.202251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:07.202195 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" podStartSLOduration=2.756365453 podStartE2EDuration="3.202162705s" totalImagesPullingTime="445.797252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:07.20016924 +0000 UTC m=+986.022765404" watchObservedRunningTime="2026-11-03 23:46:07.202162705 +0000 UTC m=+986.024758869" Nov 03 23:46:07.878201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:07.878167 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:46:07.879257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:07.879230 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:46:07.879883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:07.879856 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:46:07.898305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:07.898275 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:46:07.899277 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:07.899257 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:46:08.062869 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:08.062820 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:08.161555 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:08.161473 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:08.814192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:08.814157 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:08.914016 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:08.913984 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:09.281945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:09.281893 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:09.627896 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:09.627787 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:09.832359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:09.832318 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:10.609786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:10.609748 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:10.732885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:10.732848 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:11.342071 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:11.342003 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:12.050077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.050024 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8"] Nov 03 23:46:12.055567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.055546 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.062851 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.062811 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 23:46:12.062851 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.062827 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 23:46:12.062851 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.062814 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 23:46:12.063177 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.062917 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q44ht\"" Nov 03 23:46:12.069226 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.069203 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8"] Nov 03 23:46:12.189432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.189386 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b086dba5-92ce-444f-b264-38744cca67f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.189624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.189540 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.189624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.189580 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.189727 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.189705 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.189760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.189737 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rw6r\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-kube-api-access-6rw6r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.190003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.189976 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.190157 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.190138 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.306190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.302052 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.306190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.302168 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.306190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.302230 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b086dba5-92ce-444f-b264-38744cca67f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.306190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.302275 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.306190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.302302 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.306190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.302407 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.306190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.302434 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rw6r\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-kube-api-access-6rw6r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.307553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.307518 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.335449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.334607 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rw6r\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-kube-api-access-6rw6r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.335718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.335647 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.338350 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.338193 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.338911 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.338885 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:12.344746 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.344721 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.347415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.347393 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b086dba5-92ce-444f-b264-38744cca67f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.352049 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.348601 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-92wb8\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.376648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.375320 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:12.710147 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:46:12.710110 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb086dba5_92ce_444f_b264_38744cca67f5.slice/crio-df4d2bef45f6b2c73c96211cff2f3b6c7fd97e5f4126f9f6cff69ac121ec943a WatchSource:0}: Error finding container df4d2bef45f6b2c73c96211cff2f3b6c7fd97e5f4126f9f6cff69ac121ec943a: Status 404 returned error can't find the container with id df4d2bef45f6b2c73c96211cff2f3b6c7fd97e5f4126f9f6cff69ac121ec943a Nov 03 23:46:12.713548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.713519 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8"] Nov 03 23:46:12.839921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:12.839871 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:13.099477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:13.099270 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:13.193143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:13.193109 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:13.440465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:13.440360 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" event={"ID":"b086dba5-92ce-444f-b264-38744cca67f5","Type":"ContainerStarted","Data":"df4d2bef45f6b2c73c96211cff2f3b6c7fd97e5f4126f9f6cff69ac121ec943a"} Nov 03 23:46:13.645057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:13.642095 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:14.506634 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:14.506593 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:14.617531 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:14.617491 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:14.862413 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:14.862320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:15.854979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:15.854945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:16.483675 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:16.483645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:16.831150 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:16.831063 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:17.334268 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:17.334234 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:17.334673 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:17.334306 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:17.475381 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:17.475338 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" event={"ID":"b086dba5-92ce-444f-b264-38744cca67f5","Type":"ContainerStarted","Data":"0af1ace1c32fad9225368d86714d42bb01e2b13d9e36df0d13deaf8d6845fdeb"} Nov 03 23:46:17.475540 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:17.475470 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:17.505351 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:17.505296 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" podStartSLOduration=0.884840283 podStartE2EDuration="5.505281879s" totalImagesPullingTime="4.620441596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:46:17.502508486 +0000 UTC m=+996.325104648" watchObservedRunningTime="2026-11-03 23:46:17.505281879 +0000 UTC m=+996.327878040" Nov 03 23:46:17.853362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:17.853326 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:18.484882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:18.484853 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 03 23:46:18.763916 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:18.763822 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:19.633739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:19.633703 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:19.851845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:19.851760 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:20.687956 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:20.687916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:21.583573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:21.583542 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:22.412461 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:22.412424 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:22.514011 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:22.513977 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:22.618856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:22.618815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:23.469461 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:23.469428 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:24.232969 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:24.232928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:25.025187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:25.025153 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:25.729939 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:25.729870 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc"] Nov 03 23:46:25.738799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:25.738769 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg"] Nov 03 23:46:25.787599 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:25.787553 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:25.883911 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:25.883867 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:25.999115 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:25.998945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:26.607823 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:26.607791 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:26.924222 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:26.924139 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:27.832265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:27.832233 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:28.448232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:28.448192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:28.825986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:28.825949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:29.730776 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:29.730740 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:30.477263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:30.477224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:30.730923 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:30.730824 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" podUID="77e0aea2-0333-4c9a-b703-35a9f4bb7473" containerName="payload-processing" containerID="cri-o://8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6" gracePeriod=25 Nov 03 23:46:30.739886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:30.739853 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" podUID="55dfb50d-e44a-49ba-946e-7cd19ef1a513" containerName="payload-pre-processing" containerID="cri-o://f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33" gracePeriod=25 Nov 03 23:46:31.034579 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.034554 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:46:31.063401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.063372 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:46:31.159050 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.159006 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55dfb50d-e44a-49ba-946e-7cd19ef1a513-kube-api-access-6xgd9\" (UniqueName: \"kubernetes.io/projected/55dfb50d-e44a-49ba-946e-7cd19ef1a513-kube-api-access-6xgd9\") pod \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\" (UID: \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\") " Nov 03 23:46:31.159230 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.159087 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55dfb50d-e44a-49ba-946e-7cd19ef1a513-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dfb50d-e44a-49ba-946e-7cd19ef1a513-plugins-config-volume\") pod \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\" (UID: \"55dfb50d-e44a-49ba-946e-7cd19ef1a513\") " Nov 03 23:46:31.159230 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.159214 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e0aea2-0333-4c9a-b703-35a9f4bb7473-kube-api-access-568ql\" (UniqueName: \"kubernetes.io/projected/77e0aea2-0333-4c9a-b703-35a9f4bb7473-kube-api-access-568ql\") pod \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\" (UID: \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\") " Nov 03 23:46:31.159331 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.159256 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e0aea2-0333-4c9a-b703-35a9f4bb7473-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e0aea2-0333-4c9a-b703-35a9f4bb7473-plugins-config-volume\") pod \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\" (UID: \"77e0aea2-0333-4c9a-b703-35a9f4bb7473\") " Nov 03 23:46:31.159491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.159458 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55dfb50d-e44a-49ba-946e-7cd19ef1a513-plugins-config-volume" pod "55dfb50d-e44a-49ba-946e-7cd19ef1a513" (UID: "55dfb50d-e44a-49ba-946e-7cd19ef1a513"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:46:31.159700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.159677 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e0aea2-0333-4c9a-b703-35a9f4bb7473-plugins-config-volume" pod "77e0aea2-0333-4c9a-b703-35a9f4bb7473" (UID: "77e0aea2-0333-4c9a-b703-35a9f4bb7473"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:46:31.159859 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.159845 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77e0aea2-0333-4c9a-b703-35a9f4bb7473-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:31.159900 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.159865 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55dfb50d-e44a-49ba-946e-7cd19ef1a513-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:31.161656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.161634 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e0aea2-0333-4c9a-b703-35a9f4bb7473-kube-api-access-568ql" pod "77e0aea2-0333-4c9a-b703-35a9f4bb7473" (UID: "77e0aea2-0333-4c9a-b703-35a9f4bb7473"). InnerVolumeSpecName "kube-api-access-568ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:46:31.161743 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.161688 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55dfb50d-e44a-49ba-946e-7cd19ef1a513-kube-api-access-6xgd9" pod "55dfb50d-e44a-49ba-946e-7cd19ef1a513" (UID: "55dfb50d-e44a-49ba-946e-7cd19ef1a513"). InnerVolumeSpecName "kube-api-access-6xgd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:46:31.261643 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.261561 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xgd9\" (UniqueName: \"kubernetes.io/projected/55dfb50d-e44a-49ba-946e-7cd19ef1a513-kube-api-access-6xgd9\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:31.261643 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.261590 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-568ql\" (UniqueName: \"kubernetes.io/projected/77e0aea2-0333-4c9a-b703-35a9f4bb7473-kube-api-access-568ql\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:31.268552 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.268511 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:31.579745 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.579656 6208 generic.go:356] "Generic (PLEG): container finished" podID="77e0aea2-0333-4c9a-b703-35a9f4bb7473" containerID="8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6" exitCode=0 Nov 03 23:46:31.579745 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.579701 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" event={"ID":"77e0aea2-0333-4c9a-b703-35a9f4bb7473","Type":"ContainerDied","Data":"8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6"} Nov 03 23:46:31.579745 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.579736 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" Nov 03 23:46:31.580015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.579757 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc" event={"ID":"77e0aea2-0333-4c9a-b703-35a9f4bb7473","Type":"ContainerDied","Data":"15f6d579a90b63b2caf018e9264298cf71ead8c304ea6f3f0175c6412b7d2cbe"} Nov 03 23:46:31.580015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.579783 6208 scope.go:122] "RemoveContainer" containerID="8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6" Nov 03 23:46:31.581271 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.581247 6208 generic.go:356] "Generic (PLEG): container finished" podID="55dfb50d-e44a-49ba-946e-7cd19ef1a513" containerID="f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33" exitCode=0 Nov 03 23:46:31.581406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.581279 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" event={"ID":"55dfb50d-e44a-49ba-946e-7cd19ef1a513","Type":"ContainerDied","Data":"f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33"} Nov 03 23:46:31.581406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.581291 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" Nov 03 23:46:31.581406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.581332 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg" event={"ID":"55dfb50d-e44a-49ba-946e-7cd19ef1a513","Type":"ContainerDied","Data":"10a143a8c8f1a813022be4c3ac14443c364ea0f3dd9049cb61f00fde407cd57d"} Nov 03 23:46:31.594920 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.594896 6208 scope.go:122] "RemoveContainer" containerID="8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6" Nov 03 23:46:31.595387 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:46:31.595363 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6\": container with ID starting with 8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6 not found: ID does not exist" containerID="8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6" Nov 03 23:46:31.595504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.595397 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6"} err="failed to get container status \"8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6\": rpc error: code = NotFound desc = could not find container \"8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6\": container with ID starting with 8ee91875d2f809777e7b1a080205dafb9ad255114580fe42872687377cffcaf6 not found: ID does not exist" Nov 03 23:46:31.595504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.595420 6208 scope.go:122] "RemoveContainer" containerID="f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33" Nov 03 23:46:31.602155 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.602130 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg"] Nov 03 23:46:31.605927 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.605905 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-344261-84bfd9587f-zpvtg"] Nov 03 23:46:31.606430 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.606415 6208 scope.go:122] "RemoveContainer" containerID="f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33" Nov 03 23:46:31.606747 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:46:31.606723 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33\": container with ID starting with f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33 not found: ID does not exist" containerID="f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33" Nov 03 23:46:31.606796 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.606756 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33"} err="failed to get container status \"f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33\": rpc error: code = NotFound desc = could not find container \"f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33\": container with ID starting with f3bf6ff214f036c168fd19ff82043e2f61b7b768b4dd0eb83125fc4f89838a33 not found: ID does not exist" Nov 03 23:46:31.617321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.617297 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc"] Nov 03 23:46:31.622964 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:31.622944 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-344261-7cd5d4c778-7f6hc"] Nov 03 23:46:32.122503 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:32.122463 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:33.002454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:33.002417 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:33.186720 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:33.186686 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:33.296382 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:33.296305 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55dfb50d-e44a-49ba-946e-7cd19ef1a513" path="/var/lib/kubelet/pods/55dfb50d-e44a-49ba-946e-7cd19ef1a513/volumes" Nov 03 23:46:33.296802 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:33.296788 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e0aea2-0333-4c9a-b703-35a9f4bb7473" path="/var/lib/kubelet/pods/77e0aea2-0333-4c9a-b703-35a9f4bb7473/volumes" Nov 03 23:46:34.087919 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:34.087834 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:34.218062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:34.212742 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:34.646063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:34.645251 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:35.040635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:35.040602 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:35.629638 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:35.629601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:35.793574 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:46:35.793531 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:46:35.823085 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:35.823016 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:36.507245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:36.507210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:37.333958 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:37.333920 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:38.241768 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:38.241725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:39.017297 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:39.017262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:39.511637 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:39.509708 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:39.609980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:39.609945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:40.038804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:40.038773 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:40.873618 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:40.873490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:41.327878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:41.327820 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:41.522656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:41.522601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:41.621662 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:41.621575 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:41.969532 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:41.969493 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:42.439161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:42.439062 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:43.122329 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.117960 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq"] Nov 03 23:46:43.122329 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.118265 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" podUID="2ad0349e-c7a9-4b96-92e7-7955e6cceeed" containerName="maas-api" containerID="cri-o://d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5" gracePeriod=30 Nov 03 23:46:43.160770 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.160688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:43.187828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.187793 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" podUID="2ad0349e-c7a9-4b96-92e7-7955e6cceeed" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.101:8443/health\": dial tcp 10.133.0.101:8443: connect: connection refused" Nov 03 23:46:43.317734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.317695 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz"] Nov 03 23:46:43.317959 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.317931 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" podUID="21a6df0b-8978-4766-a057-f4e8c3993146" containerName="maas-api" containerID="cri-o://b307704e1064e3d4397f1dffa3d1e261c95b043840a8e70f14cbd1d7b960b587" gracePeriod=30 Nov 03 23:46:43.320703 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.320658 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:43.462296 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.462265 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:43.511934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.511857 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-openshift-service-ca\") pod \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " Nov 03 23:46:43.511934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.511933 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-kube-api-access-b29n5\" (UniqueName: \"kubernetes.io/projected/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-kube-api-access-b29n5\") pod \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " Nov 03 23:46:43.513365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.512148 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-metrics-certs\") pod \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " Nov 03 23:46:43.513365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.512426 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-maas-api-tls\") pod \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\" (UID: \"2ad0349e-c7a9-4b96-92e7-7955e6cceeed\") " Nov 03 23:46:43.559730 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.559073 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-openshift-service-ca" pod "2ad0349e-c7a9-4b96-92e7-7955e6cceeed" (UID: "2ad0349e-c7a9-4b96-92e7-7955e6cceeed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:46:43.561639 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.560817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:43.588070 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.584290 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:43.618961 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.618779 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-metrics-certs" pod "2ad0349e-c7a9-4b96-92e7-7955e6cceeed" (UID: "2ad0349e-c7a9-4b96-92e7-7955e6cceeed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:46:43.621253 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.619486 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-kube-api-access-b29n5" pod "2ad0349e-c7a9-4b96-92e7-7955e6cceeed" (UID: "2ad0349e-c7a9-4b96-92e7-7955e6cceeed"). InnerVolumeSpecName "kube-api-access-b29n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:46:43.621253 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.619595 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-maas-api-tls" pod "2ad0349e-c7a9-4b96-92e7-7955e6cceeed" (UID: "2ad0349e-c7a9-4b96-92e7-7955e6cceeed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:46:43.694115 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.692618 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:43.694115 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.692656 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b29n5\" (UniqueName: \"kubernetes.io/projected/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-kube-api-access-b29n5\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:43.694115 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.692672 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ad0349e-c7a9-4b96-92e7-7955e6cceeed-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:43.868077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.867860 6208 generic.go:356] "Generic (PLEG): container finished" podID="21a6df0b-8978-4766-a057-f4e8c3993146" containerID="b307704e1064e3d4397f1dffa3d1e261c95b043840a8e70f14cbd1d7b960b587" exitCode=0 Nov 03 23:46:43.868077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.868053 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" event={"ID":"21a6df0b-8978-4766-a057-f4e8c3993146","Type":"ContainerDied","Data":"b307704e1064e3d4397f1dffa3d1e261c95b043840a8e70f14cbd1d7b960b587"} Nov 03 23:46:43.871060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.870003 6208 generic.go:356] "Generic (PLEG): container finished" podID="2ad0349e-c7a9-4b96-92e7-7955e6cceeed" containerID="d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5" exitCode=0 Nov 03 23:46:43.871060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.870149 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" event={"ID":"2ad0349e-c7a9-4b96-92e7-7955e6cceeed","Type":"ContainerDied","Data":"d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5"} Nov 03 23:46:43.871060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.870193 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" event={"ID":"2ad0349e-c7a9-4b96-92e7-7955e6cceeed","Type":"ContainerDied","Data":"e4936d024644e708f413558c71a9a781336032c720f98e97fb0daf69f5416989"} Nov 03 23:46:43.871060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.870226 6208 scope.go:122] "RemoveContainer" containerID="d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5" Nov 03 23:46:43.871060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.870391 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq" Nov 03 23:46:43.899476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.898364 6208 scope.go:122] "RemoveContainer" containerID="d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5" Nov 03 23:46:43.899476 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:46:43.899058 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5\": container with ID starting with d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5 not found: ID does not exist" containerID="d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5" Nov 03 23:46:43.899476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.899096 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5"} err="failed to get container status \"d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5\": rpc error: code = NotFound desc = could not find container \"d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5\": container with ID starting with d0fc9c988255d58e371b06b696fadb2b6a9503528d251b5e2634dde3d50dc5f5 not found: ID does not exist" Nov 03 23:46:43.917564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.917532 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq"] Nov 03 23:46:43.918880 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.918850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:43.921490 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.921464 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:46:43.922367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:43.922340 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc7bd0-5f9547cbf6-dp2fq"] Nov 03 23:46:44.100301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.100260 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-maas-api-tls\") pod \"21a6df0b-8978-4766-a057-f4e8c3993146\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " Nov 03 23:46:44.100502 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.100365 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-metrics-certs\") pod \"21a6df0b-8978-4766-a057-f4e8c3993146\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " Nov 03 23:46:44.100584 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.100502 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a6df0b-8978-4766-a057-f4e8c3993146-kube-api-access-cjlvt\" (UniqueName: \"kubernetes.io/projected/21a6df0b-8978-4766-a057-f4e8c3993146-kube-api-access-cjlvt\") pod \"21a6df0b-8978-4766-a057-f4e8c3993146\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " Nov 03 23:46:44.100584 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.100543 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a6df0b-8978-4766-a057-f4e8c3993146-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a6df0b-8978-4766-a057-f4e8c3993146-openshift-service-ca\") pod \"21a6df0b-8978-4766-a057-f4e8c3993146\" (UID: \"21a6df0b-8978-4766-a057-f4e8c3993146\") " Nov 03 23:46:44.101028 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.100996 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a6df0b-8978-4766-a057-f4e8c3993146-openshift-service-ca" pod "21a6df0b-8978-4766-a057-f4e8c3993146" (UID: "21a6df0b-8978-4766-a057-f4e8c3993146"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:46:44.101358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.101333 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a6df0b-8978-4766-a057-f4e8c3993146-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:44.102563 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.102530 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-maas-api-tls" pod "21a6df0b-8978-4766-a057-f4e8c3993146" (UID: "21a6df0b-8978-4766-a057-f4e8c3993146"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:46:44.102649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.102554 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-metrics-certs" pod "21a6df0b-8978-4766-a057-f4e8c3993146" (UID: "21a6df0b-8978-4766-a057-f4e8c3993146"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:46:44.102649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.102588 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a6df0b-8978-4766-a057-f4e8c3993146-kube-api-access-cjlvt" pod "21a6df0b-8978-4766-a057-f4e8c3993146" (UID: "21a6df0b-8978-4766-a057-f4e8c3993146"). InnerVolumeSpecName "kube-api-access-cjlvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:46:44.203275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.203235 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:44.203275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.203274 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a6df0b-8978-4766-a057-f4e8c3993146-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:44.203515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.203292 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjlvt\" (UniqueName: \"kubernetes.io/projected/21a6df0b-8978-4766-a057-f4e8c3993146-kube-api-access-cjlvt\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:44.708388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.708348 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:44.879737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.879707 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" event={"ID":"21a6df0b-8978-4766-a057-f4e8c3993146","Type":"ContainerDied","Data":"53bf04fb13669727d2fd030af4b053eea34bc81853c84d44f049b80477a090d4"} Nov 03 23:46:45.262440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.879749 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz" Nov 03 23:46:45.262440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.879762 6208 scope.go:122] "RemoveContainer" containerID="b307704e1064e3d4397f1dffa3d1e261c95b043840a8e70f14cbd1d7b960b587" Nov 03 23:46:45.262440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.922097 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz"] Nov 03 23:46:45.262440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:44.925259 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-953960-567f9c684-5dqlz"] Nov 03 23:46:45.297147 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:45.297115 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a6df0b-8978-4766-a057-f4e8c3993146" path="/var/lib/kubelet/pods/21a6df0b-8978-4766-a057-f4e8c3993146/volumes" Nov 03 23:46:45.297574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:45.297559 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad0349e-c7a9-4b96-92e7-7955e6cceeed" path="/var/lib/kubelet/pods/2ad0349e-c7a9-4b96-92e7-7955e6cceeed/volumes" Nov 03 23:46:45.458312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:45.458279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:45.888221 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:45.888188 6208 generic.go:356] "Generic (PLEG): container finished" podID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerID="d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0" exitCode=0 Nov 03 23:46:45.888678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:45.888260 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" event={"ID":"3350a74c-ab8f-49db-957c-f32d1dff3241","Type":"ContainerDied","Data":"d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0"} Nov 03 23:46:45.888750 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:45.888714 6208 scope.go:122] "RemoveContainer" containerID="d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0" Nov 03 23:46:45.890294 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:45.890261 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:45.890405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:45.890328 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:45.890405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:45.890372 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:46:46.219981 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:46.219948 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:46.900367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:46.900326 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" event={"ID":"3350a74c-ab8f-49db-957c-f32d1dff3241","Type":"ContainerStarted","Data":"5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989"} Nov 03 23:46:46.979026 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:46.978992 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:47.329757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:47.329726 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:46:47.329960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:47.329940 6208 patch_prober.go:29] interesting pod/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 23:46:47.330017 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:47.329983 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 23:46:47.978076 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:47.977360 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:48.329895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:48.329809 6208 patch_prober.go:29] interesting pod/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 23:46:48.329895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:48.329863 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 23:46:48.850908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:48.850867 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:49.178112 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:49.177097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:49.302000 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:49.301964 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:49.329999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:49.329897 6208 patch_prober.go:29] interesting pod/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 23:46:49.329999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:49.329954 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 23:46:49.611143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:49.611109 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:49.816025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:49.815984 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:49.912362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:49.912279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:50.207726 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:50.207694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:50.340352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:50.340316 6208 patch_prober.go:29] interesting pod/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 23:46:50.340636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:50.340604 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 23:46:50.778590 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:50.778548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:50.887114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:50.887075 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:50.994589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:50.994555 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:51.292685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:51.292651 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:51.329676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:51.329643 6208 patch_prober.go:29] interesting pod/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 23:46:51.329865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:51.329715 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 23:46:51.611995 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:51.611920 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc"] Nov 03 23:46:51.612186 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:51.612135 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" containerID="cri-o://5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989" gracePeriod=30 Nov 03 23:46:51.679968 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:51.679929 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:51.866985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:51.866180 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:52.738020 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:52.737981 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:53.460456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:53.460422 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:54.317115 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:54.317076 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:55.032915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:55.032882 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:46:55.357078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:55.356965 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:55.714801 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:55.714749 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:56.270743 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:56.270704 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:57.254542 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.254455 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:57.481305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.480877 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:46:57.561958 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.561923 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-data\") pod \"3350a74c-ab8f-49db-957c-f32d1dff3241\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " Nov 03 23:46:57.562389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.562362 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-certs\") pod \"3350a74c-ab8f-49db-957c-f32d1dff3241\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " Nov 03 23:46:57.562808 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.562787 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-kube-api-access-bqz9g\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-kube-api-access-bqz9g\") pod \"3350a74c-ab8f-49db-957c-f32d1dff3241\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " Nov 03 23:46:57.562997 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.562978 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-socket\") pod \"3350a74c-ab8f-49db-957c-f32d1dff3241\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " Nov 03 23:46:57.563701 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.563677 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3350a74c-ab8f-49db-957c-f32d1dff3241-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3350a74c-ab8f-49db-957c-f32d1dff3241-istiod-ca-cert\") pod \"3350a74c-ab8f-49db-957c-f32d1dff3241\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " Nov 03 23:46:57.564328 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.564306 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-credential-socket\") pod \"3350a74c-ab8f-49db-957c-f32d1dff3241\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " Nov 03 23:46:57.566286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.566263 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-token\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-token\") pod \"3350a74c-ab8f-49db-957c-f32d1dff3241\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " Nov 03 23:46:57.569137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.569111 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-envoy\") pod \"3350a74c-ab8f-49db-957c-f32d1dff3241\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " Nov 03 23:46:57.571194 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.571171 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-podinfo\") pod \"3350a74c-ab8f-49db-957c-f32d1dff3241\" (UID: \"3350a74c-ab8f-49db-957c-f32d1dff3241\") " Nov 03 23:46:57.578548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.563858 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-data" pod "3350a74c-ab8f-49db-957c-f32d1dff3241" (UID: "3350a74c-ab8f-49db-957c-f32d1dff3241"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:46:57.582404 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.571503 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-kube-api-access-bqz9g" pod "3350a74c-ab8f-49db-957c-f32d1dff3241" (UID: "3350a74c-ab8f-49db-957c-f32d1dff3241"). InnerVolumeSpecName "kube-api-access-bqz9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:46:57.582670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.578515 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-certs" pod "3350a74c-ab8f-49db-957c-f32d1dff3241" (UID: "3350a74c-ab8f-49db-957c-f32d1dff3241"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:46:57.587134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.587107 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-socket" pod "3350a74c-ab8f-49db-957c-f32d1dff3241" (UID: "3350a74c-ab8f-49db-957c-f32d1dff3241"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:46:57.591679 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.591652 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3350a74c-ab8f-49db-957c-f32d1dff3241-istiod-ca-cert" pod "3350a74c-ab8f-49db-957c-f32d1dff3241" (UID: "3350a74c-ab8f-49db-957c-f32d1dff3241"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:46:57.601573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.601520 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-credential-socket" pod "3350a74c-ab8f-49db-957c-f32d1dff3241" (UID: "3350a74c-ab8f-49db-957c-f32d1dff3241"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:46:57.605028 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.604980 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-podinfo" pod "3350a74c-ab8f-49db-957c-f32d1dff3241" (UID: "3350a74c-ab8f-49db-957c-f32d1dff3241"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:46:57.607063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.607014 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-envoy" pod "3350a74c-ab8f-49db-957c-f32d1dff3241" (UID: "3350a74c-ab8f-49db-957c-f32d1dff3241"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:46:57.607379 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.607358 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-token" pod "3350a74c-ab8f-49db-957c-f32d1dff3241" (UID: "3350a74c-ab8f-49db-957c-f32d1dff3241"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:46:57.678957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.678870 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:57.679248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.679222 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:57.679363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.679349 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:57.679452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.679438 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqz9g\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-kube-api-access-bqz9g\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:57.679535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.679523 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:57.679625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.679612 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3350a74c-ab8f-49db-957c-f32d1dff3241-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:57.679706 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.679695 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:57.679786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.679776 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:57.679868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:57.679854 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3350a74c-ab8f-49db-957c-f32d1dff3241-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:46:58.027837 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.027794 6208 generic.go:356] "Generic (PLEG): container finished" podID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerID="5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989" exitCode=0 Nov 03 23:46:58.028260 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.028233 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" event={"ID":"3350a74c-ab8f-49db-957c-f32d1dff3241","Type":"ContainerDied","Data":"5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989"} Nov 03 23:46:58.028484 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.028463 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" event={"ID":"3350a74c-ab8f-49db-957c-f32d1dff3241","Type":"ContainerDied","Data":"5a13c32931e02aaec6a5cbfe6e2340ba02e72f9c82c2900ab81ed5fb0c16c58d"} Nov 03 23:46:58.028622 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.028608 6208 scope.go:122] "RemoveContainer" containerID="5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989" Nov 03 23:46:58.033060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.028873 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc" Nov 03 23:46:58.093515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.088383 6208 scope.go:122] "RemoveContainer" containerID="d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0" Nov 03 23:46:58.109515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.109489 6208 scope.go:122] "RemoveContainer" containerID="5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989" Nov 03 23:46:58.110465 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:46:58.110435 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989\": container with ID starting with 5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989 not found: ID does not exist" containerID="5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989" Nov 03 23:46:58.110699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.110641 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989"} err="failed to get container status \"5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989\": rpc error: code = NotFound desc = could not find container \"5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989\": container with ID starting with 5838ec1547c37458d456d1c5e3de1df71bc8e2eead9eb34c9d3426cb705d7989 not found: ID does not exist" Nov 03 23:46:58.110841 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.110825 6208 scope.go:122] "RemoveContainer" containerID="d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0" Nov 03 23:46:58.113839 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:46:58.113811 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0\": container with ID starting with d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0 not found: ID does not exist" containerID="d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0" Nov 03 23:46:58.113992 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.113968 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0"} err="failed to get container status \"d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0\": rpc error: code = NotFound desc = could not find container \"d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0\": container with ID starting with d330461e0dd0515a4bc9b8f94cf3c390f2d4220d778e201553c92bb56c852cf0 not found: ID does not exist" Nov 03 23:46:58.147487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.147444 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc"] Nov 03 23:46:58.147762 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.147742 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cc7bd0-openshift-default-844d57d777-hfvxc"] Nov 03 23:46:58.435896 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:58.435801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:59.340732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:59.340697 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" path="/var/lib/kubelet/pods/3350a74c-ab8f-49db-957c-f32d1dff3241/volumes" Nov 03 23:46:59.725455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:59.725175 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:46:59.835672 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:46:59.835633 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:00.562614 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:00.562574 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:00.772469 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:00.772430 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:01.641586 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:01.641551 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:02.435191 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:02.435156 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:03.075397 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:03.075361 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:03.433490 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:03.433392 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:04.224636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:04.224601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:05.025986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:05.025943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:05.711951 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:47:05.711912 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:47:05.712542 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:05.712519 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:05.841292 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:05.841258 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:06.635433 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:06.635391 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:07.432205 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:07.432168 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:07.532958 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:07.532897 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:07.633681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:07.633640 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:08.159683 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:08.159648 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:08.510975 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:08.510945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:08.620482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:08.620448 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:09.556608 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:09.556572 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:10.557525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:10.557489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:11.435257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:11.435225 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:12.284201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:12.284165 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:13.043682 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:13.043646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:13.851535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:13.851501 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:14.875951 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:14.875909 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:15.794505 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:15.794470 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:16.641620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:16.641566 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:17.881686 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:17.881648 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:18.841928 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:18.841888 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:19.637162 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:19.637129 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:20.395965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:20.395922 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:21.157163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:21.157126 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:21.944882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:21.944849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:22.923247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:22.923215 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:23.757834 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:23.757797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:24.543257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:24.543224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:25.332099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:25.332060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:26.141768 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:26.141736 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:26.944099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:26.944062 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:27.545941 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:27.545898 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d"] Nov 03 23:47:27.613171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:27.613138 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:27.717752 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:27.717722 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:27.990799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:27.990767 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:28.094579 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:28.094541 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:28.483701 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:28.483655 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:28.637150 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:28.637117 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:29.157760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:29.157722 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:29.941863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:29.941832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:30.750274 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:30.750244 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:31.598695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:31.598661 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:32.541826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:32.541790 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:32.546588 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:32.546557 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" podUID="e96be928-44af-4ce7-9cae-45d51a370e8f" containerName="payload-processing" containerID="cri-o://3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7" gracePeriod=25 Nov 03 23:47:32.659855 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:32.659816 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:32.938361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:32.938335 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:47:33.082247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.082164 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96be928-44af-4ce7-9cae-45d51a370e8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96be928-44af-4ce7-9cae-45d51a370e8f-plugins-config-volume\") pod \"e96be928-44af-4ce7-9cae-45d51a370e8f\" (UID: \"e96be928-44af-4ce7-9cae-45d51a370e8f\") " Nov 03 23:47:33.082406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.082305 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96be928-44af-4ce7-9cae-45d51a370e8f-kube-api-access-dbh6j\" (UniqueName: \"kubernetes.io/projected/e96be928-44af-4ce7-9cae-45d51a370e8f-kube-api-access-dbh6j\") pod \"e96be928-44af-4ce7-9cae-45d51a370e8f\" (UID: \"e96be928-44af-4ce7-9cae-45d51a370e8f\") " Nov 03 23:47:33.082558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.082534 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96be928-44af-4ce7-9cae-45d51a370e8f-plugins-config-volume" pod "e96be928-44af-4ce7-9cae-45d51a370e8f" (UID: "e96be928-44af-4ce7-9cae-45d51a370e8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:47:33.082849 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.082833 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96be928-44af-4ce7-9cae-45d51a370e8f-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:47:33.084415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.084382 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96be928-44af-4ce7-9cae-45d51a370e8f-kube-api-access-dbh6j" pod "e96be928-44af-4ce7-9cae-45d51a370e8f" (UID: "e96be928-44af-4ce7-9cae-45d51a370e8f"). InnerVolumeSpecName "kube-api-access-dbh6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:47:33.184129 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.184092 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbh6j\" (UniqueName: \"kubernetes.io/projected/e96be928-44af-4ce7-9cae-45d51a370e8f-kube-api-access-dbh6j\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:47:33.314482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.314447 6208 generic.go:356] "Generic (PLEG): container finished" podID="e96be928-44af-4ce7-9cae-45d51a370e8f" containerID="3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7" exitCode=0 Nov 03 23:47:33.314625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.314491 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" Nov 03 23:47:33.314625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.314532 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" event={"ID":"e96be928-44af-4ce7-9cae-45d51a370e8f","Type":"ContainerDied","Data":"3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7"} Nov 03 23:47:33.314625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.314586 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d" event={"ID":"e96be928-44af-4ce7-9cae-45d51a370e8f","Type":"ContainerDied","Data":"9530b02f9ab44aaeab63846e4a77a631045692631f3d877a6abd2c27188ff963"} Nov 03 23:47:33.314625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.314613 6208 scope.go:122] "RemoveContainer" containerID="3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7" Nov 03 23:47:33.324864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.324841 6208 scope.go:122] "RemoveContainer" containerID="3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7" Nov 03 23:47:33.325237 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:47:33.325204 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7\": container with ID starting with 3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7 not found: ID does not exist" containerID="3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7" Nov 03 23:47:33.325367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.325260 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7"} err="failed to get container status \"3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7\": rpc error: code = NotFound desc = could not find container \"3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7\": container with ID starting with 3cc6888f7da7bdf009cae5f924ed07712579707b95384a8fb47969f17c0df2a7 not found: ID does not exist" Nov 03 23:47:33.346836 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.346759 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d"] Nov 03 23:47:33.350170 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.350143 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-sg85d"] Nov 03 23:47:33.638509 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.638412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:33.855336 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.855306 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:33.976715 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:33.976687 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:34.679301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:34.679260 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:35.294922 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:35.294891 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96be928-44af-4ce7-9cae-45d51a370e8f" path="/var/lib/kubelet/pods/e96be928-44af-4ce7-9cae-45d51a370e8f/volumes" Nov 03 23:47:35.561485 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:35.561399 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:35.715652 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:47:35.715605 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:47:36.175708 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:36.175676 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:36.455284 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:36.455209 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:36.697758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:36.697722 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:36.828308 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:36.828162 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:37.469910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:37.469876 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:37.587214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:37.587170 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:37.697045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:37.696993 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:38.475305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:38.475267 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:39.353809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:39.353775 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:40.262771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:40.262734 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:40.764447 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:40.764386 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:40.894454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:40.894420 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:41.339019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:41.338990 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:41.952907 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:41.952856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:42.067717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:42.067672 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:42.388665 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:42.388632 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:43.133156 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:43.133123 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:44.034455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:44.034418 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:44.595570 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:44.595532 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5"] Nov 03 23:47:44.595826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:44.595799 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" podUID="a02c10da-b946-4464-b57a-4f6246dd0a72" containerName="maas-api" containerID="cri-o://6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533" gracePeriod=30 Nov 03 23:47:44.681170 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:47:44.681135 6208 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02c10da_b946_4464_b57a_4f6246dd0a72.slice/crio-conmon-6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533.scope\": RecentStats: unable to find data in memory cache]" Nov 03 23:47:44.684441 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:44.684410 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:44.859119 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:44.836671 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:44.976631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:44.976581 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:45.025347 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.025318 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:47:45.079975 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.079936 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02c10da-b946-4464-b57a-4f6246dd0a72-kube-api-access-68rfs\" (UniqueName: \"kubernetes.io/projected/a02c10da-b946-4464-b57a-4f6246dd0a72-kube-api-access-68rfs\") pod \"a02c10da-b946-4464-b57a-4f6246dd0a72\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " Nov 03 23:47:45.080530 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.080112 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-maas-api-tls\") pod \"a02c10da-b946-4464-b57a-4f6246dd0a72\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " Nov 03 23:47:45.080530 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.080318 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-metrics-certs\") pod \"a02c10da-b946-4464-b57a-4f6246dd0a72\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " Nov 03 23:47:45.080530 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.080356 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02c10da-b946-4464-b57a-4f6246dd0a72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02c10da-b946-4464-b57a-4f6246dd0a72-openshift-service-ca\") pod \"a02c10da-b946-4464-b57a-4f6246dd0a72\" (UID: \"a02c10da-b946-4464-b57a-4f6246dd0a72\") " Nov 03 23:47:45.081484 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.081456 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02c10da-b946-4464-b57a-4f6246dd0a72-openshift-service-ca" pod "a02c10da-b946-4464-b57a-4f6246dd0a72" (UID: "a02c10da-b946-4464-b57a-4f6246dd0a72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:47:45.084161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.084120 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02c10da-b946-4464-b57a-4f6246dd0a72-kube-api-access-68rfs" pod "a02c10da-b946-4464-b57a-4f6246dd0a72" (UID: "a02c10da-b946-4464-b57a-4f6246dd0a72"). InnerVolumeSpecName "kube-api-access-68rfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:47:45.084287 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.084158 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:45.084648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.084627 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-metrics-certs" pod "a02c10da-b946-4464-b57a-4f6246dd0a72" (UID: "a02c10da-b946-4464-b57a-4f6246dd0a72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:47:45.091587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.091557 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-maas-api-tls" pod "a02c10da-b946-4464-b57a-4f6246dd0a72" (UID: "a02c10da-b946-4464-b57a-4f6246dd0a72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:47:45.182231 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.182135 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rfs\" (UniqueName: \"kubernetes.io/projected/a02c10da-b946-4464-b57a-4f6246dd0a72-kube-api-access-68rfs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:47:45.182231 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.182171 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:47:45.182231 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.182188 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a02c10da-b946-4464-b57a-4f6246dd0a72-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:47:45.182231 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.182200 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02c10da-b946-4464-b57a-4f6246dd0a72-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:47:45.406964 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.406925 6208 generic.go:356] "Generic (PLEG): container finished" podID="a02c10da-b946-4464-b57a-4f6246dd0a72" containerID="6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533" exitCode=0 Nov 03 23:47:45.407203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.406972 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" Nov 03 23:47:45.407203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.406990 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" event={"ID":"a02c10da-b946-4464-b57a-4f6246dd0a72","Type":"ContainerDied","Data":"6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533"} Nov 03 23:47:45.407203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.407062 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5" event={"ID":"a02c10da-b946-4464-b57a-4f6246dd0a72","Type":"ContainerDied","Data":"05839bc71d3258d30ccf2884605de6cdd4317ada12f7fd0fc2dd11385110e3cb"} Nov 03 23:47:45.407203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.407093 6208 scope.go:122] "RemoveContainer" containerID="6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533" Nov 03 23:47:45.417300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.417278 6208 scope.go:122] "RemoveContainer" containerID="6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533" Nov 03 23:47:45.417562 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:47:45.417544 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533\": container with ID starting with 6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533 not found: ID does not exist" containerID="6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533" Nov 03 23:47:45.417617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.417570 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533"} err="failed to get container status \"6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533\": rpc error: code = NotFound desc = could not find container \"6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533\": container with ID starting with 6907054bcfbcb7f0426c2e2fb774639ecb570955db0d9cc644d6493f14fbc533 not found: ID does not exist" Nov 03 23:47:45.432875 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.432797 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5"] Nov 03 23:47:45.445548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:45.445510 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88f34dbe-884ff66c-p68f5"] Nov 03 23:47:46.141049 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:46.140994 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:46.338404 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:46.338367 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:47.025342 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.025299 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:47.273485 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.273446 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:47.296581 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.296495 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02c10da-b946-4464-b57a-4f6246dd0a72" path="/var/lib/kubelet/pods/a02c10da-b946-4464-b57a-4f6246dd0a72/volumes" Nov 03 23:47:47.482828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.482787 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb"] Nov 03 23:47:47.483886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.483859 6208 cpu_manager.go:414] "containerMap: removing container" podUID="2ad0349e-c7a9-4b96-92e7-7955e6cceeed" containerName="maas-api" Nov 03 23:47:47.484123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484103 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad0349e-c7a9-4b96-92e7-7955e6cceeed" containerName="maas-api" Nov 03 23:47:47.484267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484254 6208 cpu_manager.go:414] "containerMap: removing container" podUID="77e0aea2-0333-4c9a-b703-35a9f4bb7473" containerName="payload-processing" Nov 03 23:47:47.484366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484348 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e0aea2-0333-4c9a-b703-35a9f4bb7473" containerName="payload-processing" Nov 03 23:47:47.484445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484371 6208 cpu_manager.go:414] "containerMap: removing container" podUID="21a6df0b-8978-4766-a057-f4e8c3993146" containerName="maas-api" Nov 03 23:47:47.484445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484379 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a6df0b-8978-4766-a057-f4e8c3993146" containerName="maas-api" Nov 03 23:47:47.484445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484413 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" Nov 03 23:47:47.484445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484420 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" Nov 03 23:47:47.484445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484443 6208 cpu_manager.go:414] "containerMap: removing container" podUID="55dfb50d-e44a-49ba-946e-7cd19ef1a513" containerName="payload-pre-processing" Nov 03 23:47:47.484678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484451 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55dfb50d-e44a-49ba-946e-7cd19ef1a513" containerName="payload-pre-processing" Nov 03 23:47:47.484678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484479 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e96be928-44af-4ce7-9cae-45d51a370e8f" containerName="payload-processing" Nov 03 23:47:47.484678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484487 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96be928-44af-4ce7-9cae-45d51a370e8f" containerName="payload-processing" Nov 03 23:47:47.484678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484504 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" Nov 03 23:47:47.484678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484511 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" Nov 03 23:47:47.484678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484523 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a02c10da-b946-4464-b57a-4f6246dd0a72" containerName="maas-api" Nov 03 23:47:47.484678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484530 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02c10da-b946-4464-b57a-4f6246dd0a72" containerName="maas-api" Nov 03 23:47:47.484998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484708 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" Nov 03 23:47:47.484998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484720 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02c10da-b946-4464-b57a-4f6246dd0a72" containerName="maas-api" Nov 03 23:47:47.484998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484737 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad0349e-c7a9-4b96-92e7-7955e6cceeed" containerName="maas-api" Nov 03 23:47:47.484998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484749 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e0aea2-0333-4c9a-b703-35a9f4bb7473" containerName="payload-processing" Nov 03 23:47:47.484998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484760 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a6df0b-8978-4766-a057-f4e8c3993146" containerName="maas-api" Nov 03 23:47:47.484998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484772 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3350a74c-ab8f-49db-957c-f32d1dff3241" containerName="istio-proxy" Nov 03 23:47:47.484998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484784 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="55dfb50d-e44a-49ba-946e-7cd19ef1a513" containerName="payload-pre-processing" Nov 03 23:47:47.484998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.484798 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96be928-44af-4ce7-9cae-45d51a370e8f" containerName="payload-processing" Nov 03 23:47:47.491377 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.490301 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb"] Nov 03 23:47:47.491377 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.490415 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.519624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.507314 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e4892d-serving-cert\"" Nov 03 23:47:47.529893 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.529860 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:47.629757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.629655 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cb46f5-7aa2-463f-b606-9e9744603ce5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.630152 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.630125 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.630406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.630388 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-metrics-certs\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.630553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.630538 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2gj\" (UniqueName: \"kubernetes.io/projected/a3cb46f5-7aa2-463f-b606-9e9744603ce5-kube-api-access-9d2gj\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.662583 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.662545 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:47.732433 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.732384 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.732631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.732484 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-metrics-certs\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.732631 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.732556 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2gj\" (UniqueName: \"kubernetes.io/projected/a3cb46f5-7aa2-463f-b606-9e9744603ce5-kube-api-access-9d2gj\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.733833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.733800 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cb46f5-7aa2-463f-b606-9e9744603ce5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.733965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.733026 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cb46f5-7aa2-463f-b606-9e9744603ce5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.735209 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.735184 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.735518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.735494 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-metrics-certs\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.750051 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.750001 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2gj\" (UniqueName: \"kubernetes.io/projected/a3cb46f5-7aa2-463f-b606-9e9744603ce5-kube-api-access-9d2gj\") pod \"maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:47.774255 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.773424 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:47.880120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:47.879954 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:48.195798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:48.195765 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb"] Nov 03 23:47:48.197146 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:47:48.197113 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cb46f5_7aa2_463f_b606_9e9744603ce5.slice/crio-2d275c42513220d59ac98e4daf59cadb826054dd2fa32cd399657644d8a6b18e WatchSource:0}: Error finding container 2d275c42513220d59ac98e4daf59cadb826054dd2fa32cd399657644d8a6b18e: Status 404 returned error can't find the container with id 2d275c42513220d59ac98e4daf59cadb826054dd2fa32cd399657644d8a6b18e Nov 03 23:47:48.454169 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:48.454068 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" event={"ID":"a3cb46f5-7aa2-463f-b606-9e9744603ce5","Type":"ContainerStarted","Data":"2d275c42513220d59ac98e4daf59cadb826054dd2fa32cd399657644d8a6b18e"} Nov 03 23:47:48.651494 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:48.650526 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:48.805140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:48.804811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:49.462808 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:49.462761 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" event={"ID":"a3cb46f5-7aa2-463f-b606-9e9744603ce5","Type":"ContainerStarted","Data":"f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971"} Nov 03 23:47:49.463232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:49.462838 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:49.486896 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:49.486834 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" podStartSLOduration=2.075351649 podStartE2EDuration="2.486813059s" totalImagesPullingTime="411.46141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:47:49.483573797 +0000 UTC m=+1088.306169962" watchObservedRunningTime="2026-11-03 23:47:49.486813059 +0000 UTC m=+1088.309409233" Nov 03 23:47:49.667950 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:49.667908 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:50.743870 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:50.743835 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:51.254101 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:51.254067 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:51.368106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:51.368067 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:51.742518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:51.742481 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:52.522681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:52.522649 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:52.746414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:52.746376 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:53.060415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:53.060381 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:54.103093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:54.103060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:55.234210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:55.234181 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:55.476494 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:55.476463 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:47:55.860237 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:55.860160 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:55.960728 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:55.960673 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:56.715184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:56.715141 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:57.511698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:57.511651 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:58.353023 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:58.352994 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:58.466205 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:58.466164 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:59.198026 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:59.197995 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:47:59.335422 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:47:59.335381 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:00.253213 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:00.253182 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:00.912702 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:00.912665 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:01.018968 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:01.018925 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:01.122232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:01.122191 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:01.231674 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:01.231641 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:01.939316 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:01.939284 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:02.674569 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:02.674532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:03.480006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:03.479973 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:04.036578 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:04.036524 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:04.139305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:04.139256 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:04.735472 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:04.735437 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:05.649474 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:05.649437 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:05.715633 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:48:05.715593 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:48:05.775520 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:05.775488 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:06.531328 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:06.531262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:07.389433 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:07.389399 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:07.803383 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:07.803346 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:07.923358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:07.923309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:08.091116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:08.082902 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:08.441733 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:08.441635 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:09.480507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:09.480472 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:10.173564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:10.163177 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:10.299186 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:10.299008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:11.208218 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:11.208180 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:11.391105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:11.391072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:11.586149 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:11.586055 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:12.512944 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:12.512906 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:12.642348 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:12.642311 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:12.776556 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:12.776466 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:13.087855 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.087775 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:13.148918 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.148880 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p"] Nov 03 23:48:13.153272 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.153247 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:13.171140 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.171113 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-88f34dbe\"" Nov 03 23:48:13.172321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.172291 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p"] Nov 03 23:48:13.203497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.203467 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8"] Nov 03 23:48:13.207596 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.207572 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:13.215182 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.215160 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-88f34dbe-dockercfg-rgsvc\"" Nov 03 23:48:13.227233 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.227204 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8"] Nov 03 23:48:13.244553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.244522 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:13.260979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.260949 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjlsg\" (UniqueName: \"kubernetes.io/projected/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-kube-api-access-vjlsg\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8\" (UID: \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:13.261154 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.260996 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9af4e8-9477-41ff-889c-f57421a80750-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p\" (UID: \"ef9af4e8-9477-41ff-889c-f57421a80750\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:13.261201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.261163 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8kj\" (UniqueName: \"kubernetes.io/projected/ef9af4e8-9477-41ff-889c-f57421a80750-kube-api-access-4p8kj\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p\" (UID: \"ef9af4e8-9477-41ff-889c-f57421a80750\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:13.261371 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.261354 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-plugins-config-volume\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8\" (UID: \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:13.367083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.362808 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-plugins-config-volume\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8\" (UID: \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:13.367083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.362973 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjlsg\" (UniqueName: \"kubernetes.io/projected/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-kube-api-access-vjlsg\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8\" (UID: \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:13.367083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.363054 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9af4e8-9477-41ff-889c-f57421a80750-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p\" (UID: \"ef9af4e8-9477-41ff-889c-f57421a80750\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:13.367083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.363133 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8kj\" (UniqueName: \"kubernetes.io/projected/ef9af4e8-9477-41ff-889c-f57421a80750-kube-api-access-4p8kj\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p\" (UID: \"ef9af4e8-9477-41ff-889c-f57421a80750\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:13.367083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.365385 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-plugins-config-volume\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8\" (UID: \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:13.367083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.366399 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9af4e8-9477-41ff-889c-f57421a80750-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p\" (UID: \"ef9af4e8-9477-41ff-889c-f57421a80750\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:13.378849 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.378781 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8kj\" (UniqueName: \"kubernetes.io/projected/ef9af4e8-9477-41ff-889c-f57421a80750-kube-api-access-4p8kj\") pod \"payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p\" (UID: \"ef9af4e8-9477-41ff-889c-f57421a80750\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:13.379093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.379029 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjlsg\" (UniqueName: \"kubernetes.io/projected/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-kube-api-access-vjlsg\") pod \"payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8\" (UID: \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\") " pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:13.471055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.468446 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:13.528449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.528410 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:13.790755 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:48:13.790723 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9af4e8_9477_41ff_889c_f57421a80750.slice/crio-aa61ad4bc5a6e709474df70fc69d0f8e497787e80dc9843416ee324612c48b2b WatchSource:0}: Error finding container aa61ad4bc5a6e709474df70fc69d0f8e497787e80dc9843416ee324612c48b2b: Status 404 returned error can't find the container with id aa61ad4bc5a6e709474df70fc69d0f8e497787e80dc9843416ee324612c48b2b Nov 03 23:48:13.792202 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.792170 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p"] Nov 03 23:48:13.864536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.864504 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" event={"ID":"ef9af4e8-9477-41ff-889c-f57421a80750","Type":"ContainerStarted","Data":"aa61ad4bc5a6e709474df70fc69d0f8e497787e80dc9843416ee324612c48b2b"} Nov 03 23:48:13.902774 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.902734 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8"] Nov 03 23:48:13.904317 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:48:13.904288 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d84339_bf8a_4c2e_898c_4ba75c8f6df7.slice/crio-a32ef0a09fb9ce0157b638ec53092312df761b47ff56a04d4902f5186b7d7510 WatchSource:0}: Error finding container a32ef0a09fb9ce0157b638ec53092312df761b47ff56a04d4902f5186b7d7510: Status 404 returned error can't find the container with id a32ef0a09fb9ce0157b638ec53092312df761b47ff56a04d4902f5186b7d7510 Nov 03 23:48:13.987645 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:13.987604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:14.875653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:14.875610 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" event={"ID":"ef9af4e8-9477-41ff-889c-f57421a80750","Type":"ContainerStarted","Data":"5a86e6add744d2a796467f83d289ae3d2bf7c88728e8a64e8eea2c6b8cdec3ae"} Nov 03 23:48:14.876120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:14.875694 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:14.877383 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:14.877352 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" event={"ID":"20d84339-bf8a-4c2e-898c-4ba75c8f6df7","Type":"ContainerStarted","Data":"1d48a33fc316e78ee42a17adcc2045fae913367ef8d77c09f228970192866ab8"} Nov 03 23:48:14.877509 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:14.877408 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" event={"ID":"20d84339-bf8a-4c2e-898c-4ba75c8f6df7","Type":"ContainerStarted","Data":"a32ef0a09fb9ce0157b638ec53092312df761b47ff56a04d4902f5186b7d7510"} Nov 03 23:48:14.877509 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:14.877441 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:14.900319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:14.900273 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" podStartSLOduration=1.900258962 podStartE2EDuration="1.900258962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:14.896573194 +0000 UTC m=+1113.719169358" watchObservedRunningTime="2026-11-03 23:48:14.900258962 +0000 UTC m=+1113.722855152" Nov 03 23:48:14.929227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:14.929169 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" podStartSLOduration=1.929155954 podStartE2EDuration="1.929155954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:14.925430396 +0000 UTC m=+1113.748026558" watchObservedRunningTime="2026-11-03 23:48:14.929155954 +0000 UTC m=+1113.751752116" Nov 03 23:48:14.976734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:14.976703 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:16.051144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:16.051097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:17.237955 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:17.237922 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:18.255221 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:18.255185 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:19.238864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:19.238826 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:20.313822 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:20.313784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:21.242948 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:21.242914 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:22.062205 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:22.062172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:22.362139 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:22.362027 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:22.517118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:22.517081 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:22.669847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:22.669758 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:22.781372 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:22.781320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:22.978259 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:22.978224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:23.240215 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.240123 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:23.240878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.240725 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l"] Nov 03 23:48:23.250507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.250480 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l"] Nov 03 23:48:23.250691 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.250597 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.330470 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.330430 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c887c7-openshift-default-dockercfg-xnbxx\"" Nov 03 23:48:23.346172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.345012 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-certs\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.346172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.345173 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-socket\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.346172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.345213 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-podinfo\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.346172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.345247 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istiod-ca-cert\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.346172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.345421 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-envoy\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.346172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.345505 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phr9g\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-kube-api-access-phr9g\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.346172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.345647 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-data\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.346172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.345745 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-credential-socket\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.346172 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.345772 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-token\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.347972 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.347931 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:23.446982 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.446942 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phr9g\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-kube-api-access-phr9g\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.447234 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.447209 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-data\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.447328 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.447311 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-credential-socket\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.447387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.447342 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-token\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.447387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.447382 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-certs\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.447494 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.447431 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-socket\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.447494 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.447456 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-podinfo\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.447494 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.447488 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istiod-ca-cert\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.447969 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.447642 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-envoy\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.447969 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.447734 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-data\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.448516 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.448488 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-socket\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.449014 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.448965 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-certs\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.449752 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.449724 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istiod-ca-cert\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.449879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.449854 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-credential-socket\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.450640 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.450619 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-envoy\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.451974 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.451954 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-podinfo\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.475848 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.475806 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phr9g\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-kube-api-access-phr9g\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.479566 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.479528 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-token\") pod \"e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.572447 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.572353 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:23.584246 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.584206 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:23.691385 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.691350 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:23.919254 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.919225 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l"] Nov 03 23:48:23.920397 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:48:23.920369 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b14f1d_b748_4b8b_a43c_79a6589b84f8.slice/crio-d2e912c6092c4cbd581136f3221489cd8b17c30a7ad34c084da5e6b86fb764f9 WatchSource:0}: Error finding container d2e912c6092c4cbd581136f3221489cd8b17c30a7ad34c084da5e6b86fb764f9: Status 404 returned error can't find the container with id d2e912c6092c4cbd581136f3221489cd8b17c30a7ad34c084da5e6b86fb764f9 Nov 03 23:48:23.922953 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.922918 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:48:23.923061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.922988 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:48:23.923061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.923020 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:48:23.943553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.943513 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" event={"ID":"12b14f1d-b748-4b8b-a43c-79a6589b84f8","Type":"ContainerStarted","Data":"d2e912c6092c4cbd581136f3221489cd8b17c30a7ad34c084da5e6b86fb764f9"} Nov 03 23:48:23.944471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:23.944445 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:24.738923 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:24.738888 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:24.952337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:24.952303 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" event={"ID":"12b14f1d-b748-4b8b-a43c-79a6589b84f8","Type":"ContainerStarted","Data":"ea7d7e28d668e5f9f456c4785e3c1030535a10e06b403dbea76936a895409d84"} Nov 03 23:48:24.980166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:24.980108 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" podStartSLOduration=1.9800896799999999 podStartE2EDuration="1.98008968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:24.977843564 +0000 UTC m=+1123.800439727" watchObservedRunningTime="2026-11-03 23:48:24.98008968 +0000 UTC m=+1123.802685842" Nov 03 23:48:25.584809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:25.584782 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:25.590219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:25.590192 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:25.785467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:25.785431 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:25.886962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:25.886876 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:48:25.887267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:25.887249 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:48:25.961328 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:25.961294 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:25.962575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:25.962553 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:48:27.081913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:27.081864 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:27.484272 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:27.484236 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:27.635828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:27.635795 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:27.818772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:27.818681 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:28.288541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:28.288507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:28.738832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:28.738797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:28.890558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:28.890519 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:29.462007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.461973 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:29.497491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.497460 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd"] Nov 03 23:48:29.501829 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.501803 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:29.516566 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.516540 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a88e9c\"" Nov 03 23:48:29.519930 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.519905 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd"] Nov 03 23:48:29.545938 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.545904 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr"] Nov 03 23:48:29.552154 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.552131 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:29.559157 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.559128 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr"] Nov 03 23:48:29.566252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.566213 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a88e9c-dockercfg-qts4m\"" Nov 03 23:48:29.582838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.582807 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:29.662996 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.662962 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cf5h\" (UniqueName: \"kubernetes.io/projected/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-kube-api-access-8cf5h\") pod \"payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr\" (UID: \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:29.663216 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.663126 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dac6ee-468a-4559-876e-f4ab60e85ba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd\" (UID: \"41dac6ee-468a-4559-876e-f4ab60e85ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:29.663292 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.663247 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr\" (UID: \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:29.663353 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.663332 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vdhn\" (UniqueName: \"kubernetes.io/projected/41dac6ee-468a-4559-876e-f4ab60e85ba9-kube-api-access-9vdhn\") pod \"payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd\" (UID: \"41dac6ee-468a-4559-876e-f4ab60e85ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:29.786118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.780223 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dac6ee-468a-4559-876e-f4ab60e85ba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd\" (UID: \"41dac6ee-468a-4559-876e-f4ab60e85ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:29.786118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.780333 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr\" (UID: \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:29.786118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.780398 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vdhn\" (UniqueName: \"kubernetes.io/projected/41dac6ee-468a-4559-876e-f4ab60e85ba9-kube-api-access-9vdhn\") pod \"payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd\" (UID: \"41dac6ee-468a-4559-876e-f4ab60e85ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:29.786118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.780585 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cf5h\" (UniqueName: \"kubernetes.io/projected/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-kube-api-access-8cf5h\") pod \"payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr\" (UID: \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:29.786118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.781845 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dac6ee-468a-4559-876e-f4ab60e85ba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd\" (UID: \"41dac6ee-468a-4559-876e-f4ab60e85ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:29.788405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.788348 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr\" (UID: \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:29.818197 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.818159 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cf5h\" (UniqueName: \"kubernetes.io/projected/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-kube-api-access-8cf5h\") pod \"payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr\" (UID: \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\") " pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:29.819058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.819018 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vdhn\" (UniqueName: \"kubernetes.io/projected/41dac6ee-468a-4559-876e-f4ab60e85ba9-kube-api-access-9vdhn\") pod \"payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd\" (UID: \"41dac6ee-468a-4559-876e-f4ab60e85ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:29.874277 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:29.874242 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:30.117526 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:30.117413 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:30.170881 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:30.170836 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr"] Nov 03 23:48:30.179312 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:48:30.176610 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f71db5_a959_4b98_b7cf_6d2b6a34915f.slice/crio-991f9bead1e0efdc3ce8d8f0443bf2165030501f85733c03312dd4adfb7b27bc WatchSource:0}: Error finding container 991f9bead1e0efdc3ce8d8f0443bf2165030501f85733c03312dd4adfb7b27bc: Status 404 returned error can't find the container with id 991f9bead1e0efdc3ce8d8f0443bf2165030501f85733c03312dd4adfb7b27bc Nov 03 23:48:30.324948 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:30.324915 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:30.411502 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:30.411392 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd"] Nov 03 23:48:30.420450 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:48:30.420416 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41dac6ee_468a_4559_876e_f4ab60e85ba9.slice/crio-68620569aecfb4c64b45a809f1ee5ce8c98834af572240f04a0a4d819e726121 WatchSource:0}: Error finding container 68620569aecfb4c64b45a809f1ee5ce8c98834af572240f04a0a4d819e726121: Status 404 returned error can't find the container with id 68620569aecfb4c64b45a809f1ee5ce8c98834af572240f04a0a4d819e726121 Nov 03 23:48:31.005302 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:31.005260 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" event={"ID":"41dac6ee-468a-4559-876e-f4ab60e85ba9","Type":"ContainerStarted","Data":"56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf"} Nov 03 23:48:31.005785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:31.005320 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" event={"ID":"41dac6ee-468a-4559-876e-f4ab60e85ba9","Type":"ContainerStarted","Data":"68620569aecfb4c64b45a809f1ee5ce8c98834af572240f04a0a4d819e726121"} Nov 03 23:48:31.005785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:31.005350 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:31.007008 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:31.006980 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" event={"ID":"f4f71db5-a959-4b98-b7cf-6d2b6a34915f","Type":"ContainerStarted","Data":"b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d"} Nov 03 23:48:31.007129 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:31.007022 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" event={"ID":"f4f71db5-a959-4b98-b7cf-6d2b6a34915f","Type":"ContainerStarted","Data":"991f9bead1e0efdc3ce8d8f0443bf2165030501f85733c03312dd4adfb7b27bc"} Nov 03 23:48:31.007129 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:31.007084 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:31.025648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:31.025604 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" podStartSLOduration=2.025588767 podStartE2EDuration="2.025588767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:31.023804382 +0000 UTC m=+1129.846400543" watchObservedRunningTime="2026-11-03 23:48:31.025588767 +0000 UTC m=+1129.848184931" Nov 03 23:48:31.045311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:31.045268 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" podStartSLOduration=2.045255843 podStartE2EDuration="2.045255843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:31.04281805 +0000 UTC m=+1129.865414213" watchObservedRunningTime="2026-11-03 23:48:31.045255843 +0000 UTC m=+1129.867852013" Nov 03 23:48:31.312211 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:31.312132 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:32.243945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:32.243906 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:33.472027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:33.471990 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:34.512050 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:34.509478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:34.662062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:34.661706 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:35.057920 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:35.057881 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:35.178732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:35.178694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:35.708403 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:48:35.708364 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:48:35.819600 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:35.819564 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:36.859055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:36.858837 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:38.118107 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:38.118072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:38.221851 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:38.221817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:38.323206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:38.323169 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:38.990739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:38.990705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:39.085985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.085952 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt"] Nov 03 23:48:39.090955 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.090932 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.102456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.102426 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt"] Nov 03 23:48:39.102875 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.102855 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a88e9c-serving-cert\"" Nov 03 23:48:39.149697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.149662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:39.175691 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.175661 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-openshift-service-ca\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.175867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.175793 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-metrics-certs\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.175867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.175846 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h76t\" (UniqueName: \"kubernetes.io/projected/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-kube-api-access-4h76t\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.175978 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.175897 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-maas-api-tls\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.277879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.277781 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-maas-api-tls\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.278264 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.278236 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-openshift-service-ca\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.278609 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.278585 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-metrics-certs\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.278845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.278825 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h76t\" (UniqueName: \"kubernetes.io/projected/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-kube-api-access-4h76t\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.279405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.279373 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-openshift-service-ca\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.282498 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.282451 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-metrics-certs\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.282599 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.282576 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-maas-api-tls\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.300675 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.300513 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:39.302903 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.302875 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h76t\" (UniqueName: \"kubernetes.io/projected/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-kube-api-access-4h76t\") pod \"maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.414528 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.414500 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:39.730306 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:39.730272 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt"] Nov 03 23:48:39.730423 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:48:39.730313 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9c5e8d_abb5_4560_9aab_49c97e5ad9e2.slice/crio-fbe6e4fbf7ed7028fa6d38e8ade4382f14aa839d95b45cbee01865a2c5ee4f6e WatchSource:0}: Error finding container fbe6e4fbf7ed7028fa6d38e8ade4382f14aa839d95b45cbee01865a2c5ee4f6e: Status 404 returned error can't find the container with id fbe6e4fbf7ed7028fa6d38e8ade4382f14aa839d95b45cbee01865a2c5ee4f6e Nov 03 23:48:40.103622 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.103522 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" event={"ID":"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2","Type":"ContainerStarted","Data":"fbe6e4fbf7ed7028fa6d38e8ade4382f14aa839d95b45cbee01865a2c5ee4f6e"} Nov 03 23:48:40.233607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.233576 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:40.350655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.350611 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:40.365890 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.365824 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp"] Nov 03 23:48:40.370245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.370226 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:40.383859 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.383835 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp"] Nov 03 23:48:40.403315 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.403272 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp\" (UID: \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:40.403507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.403362 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l59z\" (UniqueName: \"kubernetes.io/projected/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-kube-api-access-2l59z\") pod \"payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp\" (UID: \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:40.480956 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.480921 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8cfc3c\"" Nov 03 23:48:40.481497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.481462 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8cfc3c-dockercfg-vws7g\"" Nov 03 23:48:40.513086 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.510897 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp\" (UID: \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:40.513086 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.510952 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l59z\" (UniqueName: \"kubernetes.io/projected/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-kube-api-access-2l59z\") pod \"payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp\" (UID: \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:40.516899 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.516128 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp\" (UID: \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:40.546365 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.544771 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l59z\" (UniqueName: \"kubernetes.io/projected/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-kube-api-access-2l59z\") pod \"payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp\" (UID: \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:40.688699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:40.688618 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:41.003084 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:48:40.998682 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bad7ff_6d9a_4fb4_9bd6_5f4be46d612a.slice/crio-ab05329fd519e78c0442375aed82b3b16d35a39e044a806230b2b6786b82e03d WatchSource:0}: Error finding container ab05329fd519e78c0442375aed82b3b16d35a39e044a806230b2b6786b82e03d: Status 404 returned error can't find the container with id ab05329fd519e78c0442375aed82b3b16d35a39e044a806230b2b6786b82e03d Nov 03 23:48:41.003084 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:41.002084 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp"] Nov 03 23:48:41.116337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:41.116292 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" event={"ID":"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2","Type":"ContainerStarted","Data":"84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13"} Nov 03 23:48:41.116572 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:41.116545 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:41.118338 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:41.118307 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" event={"ID":"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a","Type":"ContainerStarted","Data":"227bc643c7d633bb3667ac35f594a9bf52257bb6b4b0c41c201cd03678d58868"} Nov 03 23:48:41.118445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:41.118362 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" event={"ID":"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a","Type":"ContainerStarted","Data":"ab05329fd519e78c0442375aed82b3b16d35a39e044a806230b2b6786b82e03d"} Nov 03 23:48:41.118445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:41.118392 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:41.160091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:41.160007 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" podStartSLOduration=1.780906291 podStartE2EDuration="2.159985144s" totalImagesPullingTime="379.078853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:41.153179711 +0000 UTC m=+1139.975775881" watchObservedRunningTime="2026-11-03 23:48:41.159985144 +0000 UTC m=+1139.982581337" Nov 03 23:48:41.179257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:41.179203 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" podStartSLOduration=1.179188135 podStartE2EDuration="1.179188135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:41.17647836 +0000 UTC m=+1139.999074525" watchObservedRunningTime="2026-11-03 23:48:41.179188135 +0000 UTC m=+1140.001784296" Nov 03 23:48:41.396761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:41.396674 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:42.016910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:42.016883 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:48:42.017086 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:42.016946 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:48:42.232600 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:42.232563 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:43.061273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:43.061231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:43.847239 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:43.847209 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:44.650106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:44.650074 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:44.756947 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:44.756913 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:45.709652 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:45.709616 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:46.476754 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:46.476716 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:47.130227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:47.130196 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:48:47.446210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:47.446121 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:48.353656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:48.353624 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:49.483372 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.483335 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z"] Nov 03 23:48:49.489645 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.489624 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.497964 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.497938 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8cfc3c-serving-cert\"" Nov 03 23:48:49.507174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.507152 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z"] Nov 03 23:48:49.579724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.579689 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:49.648731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.648694 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.648917 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.648785 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6980ffec-ec71-4dd0-9037-d264003cbaac-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.649094 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.649065 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fq72\" (UniqueName: \"kubernetes.io/projected/6980ffec-ec71-4dd0-9037-d264003cbaac-kube-api-access-6fq72\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.649170 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.649156 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-metrics-certs\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.685953 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.685911 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:49.750748 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.750662 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fq72\" (UniqueName: \"kubernetes.io/projected/6980ffec-ec71-4dd0-9037-d264003cbaac-kube-api-access-6fq72\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.750900 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.750750 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-metrics-certs\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.750900 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.750864 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.751025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.750910 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6980ffec-ec71-4dd0-9037-d264003cbaac-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.751630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.751601 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6980ffec-ec71-4dd0-9037-d264003cbaac-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.753547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.753524 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.753718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.753701 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-metrics-certs\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.763617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.763592 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fq72\" (UniqueName: \"kubernetes.io/projected/6980ffec-ec71-4dd0-9037-d264003cbaac-kube-api-access-6fq72\") pod \"maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:49.807508 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:49.807465 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:50.097587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:50.097386 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z"] Nov 03 23:48:50.105069 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:48:50.101294 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6980ffec_ec71_4dd0_9037_d264003cbaac.slice/crio-1d66213b0785f076d9d2a1aac948a7c05ea271a8ba245bfa77eee9ad567e268e WatchSource:0}: Error finding container 1d66213b0785f076d9d2a1aac948a7c05ea271a8ba245bfa77eee9ad567e268e: Status 404 returned error can't find the container with id 1d66213b0785f076d9d2a1aac948a7c05ea271a8ba245bfa77eee9ad567e268e Nov 03 23:48:50.189761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:50.189723 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" event={"ID":"6980ffec-ec71-4dd0-9037-d264003cbaac","Type":"ContainerStarted","Data":"1d66213b0785f076d9d2a1aac948a7c05ea271a8ba245bfa77eee9ad567e268e"} Nov 03 23:48:50.811757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:50.811677 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:51.198905 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:51.198828 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" event={"ID":"6980ffec-ec71-4dd0-9037-d264003cbaac","Type":"ContainerStarted","Data":"9f790ae26feaa81e63116101adafea5bbdb6e8e82b55bd27cbd2199721cc05b9"} Nov 03 23:48:51.199091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:51.199029 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:51.220144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:51.220094 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" podStartSLOduration=1.7751463520000001 podStartE2EDuration="2.220080783s" totalImagesPullingTime="444.934431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:48:51.218689702 +0000 UTC m=+1150.041285864" watchObservedRunningTime="2026-11-03 23:48:51.220080783 +0000 UTC m=+1150.042676944" Nov 03 23:48:52.011331 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:52.011297 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:52.128709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:52.128671 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:48:53.030429 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:53.030395 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:53.222224 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:53.222193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:54.327814 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:54.327774 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:55.133271 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:55.133230 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:55.874099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:55.874022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:55.990210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:55.990008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:56.637846 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:56.637810 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:56.738191 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:56.738160 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:57.214109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:57.214077 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:48:57.572496 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:57.572410 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:58.081349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:58.081310 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:58.744069 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:58.744007 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:59.472590 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:59.472540 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:48:59.683804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:48:59.683771 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:00.445928 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:00.445894 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:01.261351 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:01.261316 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:01.600685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:01.600605 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln"] Nov 03 23:49:01.718218 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:01.718180 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:02.437473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:02.437434 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:02.580359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:02.580326 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:02.688547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:02.688432 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:03.614058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:03.613995 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:04.165243 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:04.165208 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp"] Nov 03 23:49:04.386087 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:04.372276 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:04.508001 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:04.507960 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:04.662064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:04.623467 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:05.233991 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:05.233956 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:05.710956 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:49:05.710910 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:49:05.887021 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:05.886969 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:05.988601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:05.988507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:06.540725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:06.540685 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:07.579594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:07.579556 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:07.694519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:07.694451 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:08.225146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:08.225113 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:08.612353 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:08.612269 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:09.174159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.174108 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" podUID="f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a" containerName="payload-processing" containerID="cri-o://227bc643c7d633bb3667ac35f594a9bf52257bb6b4b0c41c201cd03678d58868" gracePeriod=25 Nov 03 23:49:09.404250 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.404215 6208 generic.go:356] "Generic (PLEG): container finished" podID="f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a" containerID="227bc643c7d633bb3667ac35f594a9bf52257bb6b4b0c41c201cd03678d58868" exitCode=0 Nov 03 23:49:09.404432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.404292 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" event={"ID":"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a","Type":"ContainerDied","Data":"227bc643c7d633bb3667ac35f594a9bf52257bb6b4b0c41c201cd03678d58868"} Nov 03 23:49:09.441173 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.441094 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:09.550984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.550948 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:49:09.696387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.696303 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-kube-api-access-2l59z\" (UniqueName: \"kubernetes.io/projected/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-kube-api-access-2l59z\") pod \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\" (UID: \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\") " Nov 03 23:49:09.696843 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.696450 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-plugins-config-volume\") pod \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\" (UID: \"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a\") " Nov 03 23:49:09.696912 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.696866 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-plugins-config-volume" pod "f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a" (UID: "f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:49:09.697168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.697151 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:09.699025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.699004 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-kube-api-access-2l59z" pod "f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a" (UID: "f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a"). InnerVolumeSpecName "kube-api-access-2l59z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:49:09.798936 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:09.798899 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l59z\" (UniqueName: \"kubernetes.io/projected/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a-kube-api-access-2l59z\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:10.294171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:10.294128 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:10.415974 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:10.415939 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" Nov 03 23:49:10.416197 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:10.415967 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp" event={"ID":"f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a","Type":"ContainerDied","Data":"ab05329fd519e78c0442375aed82b3b16d35a39e044a806230b2b6786b82e03d"} Nov 03 23:49:10.416197 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:10.416055 6208 scope.go:122] "RemoveContainer" containerID="227bc643c7d633bb3667ac35f594a9bf52257bb6b4b0c41c201cd03678d58868" Nov 03 23:49:10.448804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:10.448769 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp"] Nov 03 23:49:10.452493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:10.452467 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8cfc3c-7b84544846-sszxp"] Nov 03 23:49:11.296364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:11.296331 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a" path="/var/lib/kubelet/pods/f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a/volumes" Nov 03 23:49:11.331916 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:11.331885 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:12.003199 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:12.003160 6208 patch_prober.go:29] interesting pod/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:49:12.003371 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:12.003231 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:49:12.064497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:12.064393 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:12.837380 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:12.837344 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:13.671019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:13.670970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:14.097267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:14.097234 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:14.204863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:14.204779 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:14.451066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:14.450973 6208 generic.go:356] "Generic (PLEG): container finished" podID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerID="ea7d7e28d668e5f9f456c4785e3c1030535a10e06b403dbea76936a895409d84" exitCode=0 Nov 03 23:49:14.451213 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:14.451068 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" event={"ID":"12b14f1d-b748-4b8b-a43c-79a6589b84f8","Type":"ContainerDied","Data":"ea7d7e28d668e5f9f456c4785e3c1030535a10e06b403dbea76936a895409d84"} Nov 03 23:49:14.451536 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:14.451511 6208 scope.go:122] "RemoveContainer" containerID="ea7d7e28d668e5f9f456c4785e3c1030535a10e06b403dbea76936a895409d84" Nov 03 23:49:14.453230 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:14.453199 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:49:14.453312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:14.453270 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:49:14.453312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:14.453303 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:49:14.669531 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:14.669493 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:15.078909 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:15.078872 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:15.214751 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:15.214711 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:15.473055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:15.466081 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" event={"ID":"12b14f1d-b748-4b8b-a43c-79a6589b84f8","Type":"ContainerStarted","Data":"320e8f63f3736d1476f5507ca37d7441b0542fb883b2a5a199caa27fc9b69bda"} Nov 03 23:49:15.585086 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:15.585051 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:49:15.585325 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:15.585301 6208 patch_prober.go:29] interesting pod/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 23:49:15.585380 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:15.585358 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 23:49:15.821637 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:15.821558 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:15.920773 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:15.920738 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:16.584780 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:16.584735 6208 patch_prober.go:29] interesting pod/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 23:49:16.585274 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:16.584817 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 23:49:16.623073 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:16.623012 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" podUID="6de4693f-7d97-4c22-b39c-517473ee9486" containerName="main" containerID="cri-o://ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded" gracePeriod=45 Nov 03 23:49:16.711465 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:16.711433 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:16.990383 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:16.990353 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:49:17.042493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.042452 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-dshm\") pod \"6de4693f-7d97-4c22-b39c-517473ee9486\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " Nov 03 23:49:17.042690 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.042540 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de4693f-7d97-4c22-b39c-517473ee9486-tls-certs\" (UniqueName: \"kubernetes.io/secret/6de4693f-7d97-4c22-b39c-517473ee9486-tls-certs\") pod \"6de4693f-7d97-4c22-b39c-517473ee9486\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " Nov 03 23:49:17.042690 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.042617 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-tmp-dir\") pod \"6de4693f-7d97-4c22-b39c-517473ee9486\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " Nov 03 23:49:17.042690 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.042684 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-model-cache\") pod \"6de4693f-7d97-4c22-b39c-517473ee9486\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " Nov 03 23:49:17.042858 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.042756 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de4693f-7d97-4c22-b39c-517473ee9486-kube-api-access-8wgvn\" (UniqueName: \"kubernetes.io/projected/6de4693f-7d97-4c22-b39c-517473ee9486-kube-api-access-8wgvn\") pod \"6de4693f-7d97-4c22-b39c-517473ee9486\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " Nov 03 23:49:17.042858 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.042827 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-home\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-home\") pod \"6de4693f-7d97-4c22-b39c-517473ee9486\" (UID: \"6de4693f-7d97-4c22-b39c-517473ee9486\") " Nov 03 23:49:17.045493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.045133 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-home" pod "6de4693f-7d97-4c22-b39c-517473ee9486" (UID: "6de4693f-7d97-4c22-b39c-517473ee9486"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:49:17.045937 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.045905 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-model-cache" pod "6de4693f-7d97-4c22-b39c-517473ee9486" (UID: "6de4693f-7d97-4c22-b39c-517473ee9486"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:49:17.050091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.050027 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de4693f-7d97-4c22-b39c-517473ee9486-kube-api-access-8wgvn" pod "6de4693f-7d97-4c22-b39c-517473ee9486" (UID: "6de4693f-7d97-4c22-b39c-517473ee9486"). InnerVolumeSpecName "kube-api-access-8wgvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:49:17.050667 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.050614 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-tmp-dir" pod "6de4693f-7d97-4c22-b39c-517473ee9486" (UID: "6de4693f-7d97-4c22-b39c-517473ee9486"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:49:17.061060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.053114 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-dshm" pod "6de4693f-7d97-4c22-b39c-517473ee9486" (UID: "6de4693f-7d97-4c22-b39c-517473ee9486"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:49:17.070416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.070167 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de4693f-7d97-4c22-b39c-517473ee9486-tls-certs" pod "6de4693f-7d97-4c22-b39c-517473ee9486" (UID: "6de4693f-7d97-4c22-b39c-517473ee9486"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:49:17.145478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.145399 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:17.145478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.145429 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:17.145478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.145438 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de4693f-7d97-4c22-b39c-517473ee9486-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:17.145478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.145464 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:17.145478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.145473 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de4693f-7d97-4c22-b39c-517473ee9486-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:17.145756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.145486 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgvn\" (UniqueName: \"kubernetes.io/projected/6de4693f-7d97-4c22-b39c-517473ee9486-kube-api-access-8wgvn\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:17.497603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.497563 6208 generic.go:356] "Generic (PLEG): container finished" podID="6de4693f-7d97-4c22-b39c-517473ee9486" containerID="ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded" exitCode=0 Nov 03 23:49:17.497824 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.497800 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" Nov 03 23:49:17.497917 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.497705 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" event={"ID":"6de4693f-7d97-4c22-b39c-517473ee9486","Type":"ContainerDied","Data":"ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded"} Nov 03 23:49:17.497976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.497925 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln" event={"ID":"6de4693f-7d97-4c22-b39c-517473ee9486","Type":"ContainerDied","Data":"51eb32883e624572ac9680f4b66aa96bc9e49f52e9e8c9875954bf05da6af5e7"} Nov 03 23:49:17.497976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.497955 6208 scope.go:122] "RemoveContainer" containerID="ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded" Nov 03 23:49:17.509914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.509893 6208 scope.go:122] "RemoveContainer" containerID="ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded" Nov 03 23:49:17.510254 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:49:17.510229 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded\": container with ID starting with ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded not found: ID does not exist" containerID="ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded" Nov 03 23:49:17.510319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.510261 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded"} err="failed to get container status \"ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded\": rpc error: code = NotFound desc = could not find container \"ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded\": container with ID starting with ed56e9dc97fb51d1d81c85c112dfbba102e80a0e03294f260d57c79630d64ded not found: ID does not exist" Nov 03 23:49:17.520103 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.520071 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln"] Nov 03 23:49:17.526655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.526634 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-88f34dbe/e2e-lifecycle-model-88f34dbe-kserve-6cfc644c89-j67ln"] Nov 03 23:49:17.586515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.586469 6208 patch_prober.go:29] interesting pod/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 23:49:17.586933 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:17.586541 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 23:49:18.005654 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:18.005617 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:18.585398 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:18.585365 6208 patch_prober.go:29] interesting pod/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 23:49:18.585570 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:18.585420 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 23:49:18.917517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:18.917422 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:19.296731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:19.296691 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de4693f-7d97-4c22-b39c-517473ee9486" path="/var/lib/kubelet/pods/6de4693f-7d97-4c22-b39c-517473ee9486/volumes" Nov 03 23:49:19.585522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:19.585440 6208 patch_prober.go:29] interesting pod/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 23:49:19.585522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:19.585500 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 23:49:19.791267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:19.791192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:20.660488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:20.660451 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:20.879601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:20.879555 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:49:20.880194 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:20.880166 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:49:20.881100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:20.881079 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:49:21.544313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:21.544279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:22.159353 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:22.159317 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr"] Nov 03 23:49:22.170119 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:22.170087 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd"] Nov 03 23:49:22.234723 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:22.234692 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:22.337839 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:22.337800 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:22.486471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:22.486439 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:22.941254 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:22.941181 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-4dsdb_c07df26b-e714-4555-a633-fa616bb9be2e/payload-processing/0.log" Nov 03 23:49:23.152680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:23.152654 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-4dsdb_c07df26b-e714-4555-a633-fa616bb9be2e/payload-processing/0.log" Nov 03 23:49:23.499798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:23.499767 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:23.603289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:23.603256 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:24.575879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:24.575842 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:25.483698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:25.483662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:25.591294 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:25.591261 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:26.643586 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:26.643552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:27.160169 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.160124 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" podUID="f4f71db5-a959-4b98-b7cf-6d2b6a34915f" containerName="payload-processing" containerID="cri-o://b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d" gracePeriod=25 Nov 03 23:49:27.170762 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.170729 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" podUID="41dac6ee-468a-4559-876e-f4ab60e85ba9" containerName="payload-pre-processing" containerID="cri-o://56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf" gracePeriod=25 Nov 03 23:49:27.465761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.465731 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:49:27.511282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.511251 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:49:27.540347 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.540318 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-plugins-config-volume\") pod \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\" (UID: \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\") " Nov 03 23:49:27.540555 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.540536 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-kube-api-access-8cf5h\" (UniqueName: \"kubernetes.io/projected/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-kube-api-access-8cf5h\") pod \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\" (UID: \"f4f71db5-a959-4b98-b7cf-6d2b6a34915f\") " Nov 03 23:49:27.541270 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.541236 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-plugins-config-volume" pod "f4f71db5-a959-4b98-b7cf-6d2b6a34915f" (UID: "f4f71db5-a959-4b98-b7cf-6d2b6a34915f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:49:27.544981 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.544949 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-kube-api-access-8cf5h" pod "f4f71db5-a959-4b98-b7cf-6d2b6a34915f" (UID: "f4f71db5-a959-4b98-b7cf-6d2b6a34915f"). InnerVolumeSpecName "kube-api-access-8cf5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:49:27.580477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.580445 6208 generic.go:356] "Generic (PLEG): container finished" podID="41dac6ee-468a-4559-876e-f4ab60e85ba9" containerID="56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf" exitCode=0 Nov 03 23:49:27.580649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.580489 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" Nov 03 23:49:27.580649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.580527 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" event={"ID":"41dac6ee-468a-4559-876e-f4ab60e85ba9","Type":"ContainerDied","Data":"56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf"} Nov 03 23:49:27.580649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.580583 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd" event={"ID":"41dac6ee-468a-4559-876e-f4ab60e85ba9","Type":"ContainerDied","Data":"68620569aecfb4c64b45a809f1ee5ce8c98834af572240f04a0a4d819e726121"} Nov 03 23:49:27.580649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.580609 6208 scope.go:122] "RemoveContainer" containerID="56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf" Nov 03 23:49:27.582114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.582095 6208 generic.go:356] "Generic (PLEG): container finished" podID="f4f71db5-a959-4b98-b7cf-6d2b6a34915f" containerID="b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d" exitCode=0 Nov 03 23:49:27.582210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.582145 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" event={"ID":"f4f71db5-a959-4b98-b7cf-6d2b6a34915f","Type":"ContainerDied","Data":"b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d"} Nov 03 23:49:27.582210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.582159 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" Nov 03 23:49:27.582210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.582181 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr" event={"ID":"f4f71db5-a959-4b98-b7cf-6d2b6a34915f","Type":"ContainerDied","Data":"991f9bead1e0efdc3ce8d8f0443bf2165030501f85733c03312dd4adfb7b27bc"} Nov 03 23:49:27.593114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.593096 6208 scope.go:122] "RemoveContainer" containerID="56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf" Nov 03 23:49:27.593403 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:49:27.593385 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf\": container with ID starting with 56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf not found: ID does not exist" containerID="56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf" Nov 03 23:49:27.593452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.593412 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf"} err="failed to get container status \"56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf\": rpc error: code = NotFound desc = could not find container \"56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf\": container with ID starting with 56bdac23ac3d9d23064d9b0e7a166efbb8f851d7a05891e308e88b984282d1bf not found: ID does not exist" Nov 03 23:49:27.593452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.593439 6208 scope.go:122] "RemoveContainer" containerID="b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d" Nov 03 23:49:27.602713 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.602692 6208 scope.go:122] "RemoveContainer" containerID="b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d" Nov 03 23:49:27.602968 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:49:27.602948 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d\": container with ID starting with b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d not found: ID does not exist" containerID="b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d" Nov 03 23:49:27.603021 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.602979 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d"} err="failed to get container status \"b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d\": rpc error: code = NotFound desc = could not find container \"b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d\": container with ID starting with b2e11320e38470d00f4285db8c08e962dfabf71156b10e9f560faa5fecb52b5d not found: ID does not exist" Nov 03 23:49:27.611898 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.611870 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr"] Nov 03 23:49:27.617448 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.617426 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a88e9c-7f4f5497fb-4z8kr"] Nov 03 23:49:27.642442 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.642414 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41dac6ee-468a-4559-876e-f4ab60e85ba9-kube-api-access-9vdhn\" (UniqueName: \"kubernetes.io/projected/41dac6ee-468a-4559-876e-f4ab60e85ba9-kube-api-access-9vdhn\") pod \"41dac6ee-468a-4559-876e-f4ab60e85ba9\" (UID: \"41dac6ee-468a-4559-876e-f4ab60e85ba9\") " Nov 03 23:49:27.642580 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.642458 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41dac6ee-468a-4559-876e-f4ab60e85ba9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dac6ee-468a-4559-876e-f4ab60e85ba9-plugins-config-volume\") pod \"41dac6ee-468a-4559-876e-f4ab60e85ba9\" (UID: \"41dac6ee-468a-4559-876e-f4ab60e85ba9\") " Nov 03 23:49:27.642926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.642903 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41dac6ee-468a-4559-876e-f4ab60e85ba9-plugins-config-volume" pod "41dac6ee-468a-4559-876e-f4ab60e85ba9" (UID: "41dac6ee-468a-4559-876e-f4ab60e85ba9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:49:27.643134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.643114 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cf5h\" (UniqueName: \"kubernetes.io/projected/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-kube-api-access-8cf5h\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:27.643217 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.643139 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41dac6ee-468a-4559-876e-f4ab60e85ba9-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:27.643217 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.643154 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f71db5-a959-4b98-b7cf-6d2b6a34915f-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:27.644880 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.644855 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dac6ee-468a-4559-876e-f4ab60e85ba9-kube-api-access-9vdhn" pod "41dac6ee-468a-4559-876e-f4ab60e85ba9" (UID: "41dac6ee-468a-4559-876e-f4ab60e85ba9"). InnerVolumeSpecName "kube-api-access-9vdhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:49:27.650378 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.650352 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:27.747144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.744662 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vdhn\" (UniqueName: \"kubernetes.io/projected/41dac6ee-468a-4559-876e-f4ab60e85ba9-kube-api-access-9vdhn\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:27.905455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.905420 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd"] Nov 03 23:49:27.912923 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:27.912898 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a88e9c-7bbcf7dd79-bqqsd"] Nov 03 23:49:28.639535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:28.639491 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:29.295934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:29.295898 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41dac6ee-468a-4559-876e-f4ab60e85ba9" path="/var/lib/kubelet/pods/41dac6ee-468a-4559-876e-f4ab60e85ba9/volumes" Nov 03 23:49:29.296370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:29.296353 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f71db5-a959-4b98-b7cf-6d2b6a34915f" path="/var/lib/kubelet/pods/f4f71db5-a959-4b98-b7cf-6d2b6a34915f/volumes" Nov 03 23:49:29.487866 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:29.487830 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt"] Nov 03 23:49:29.488127 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:29.488098 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" podUID="1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" containerName="maas-api" containerID="cri-o://84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13" gracePeriod=30 Nov 03 23:49:29.536525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:29.536490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:29.919953 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:29.919928 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:49:30.071642 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.071613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:30.097668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.097641 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-openshift-service-ca\") pod \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " Nov 03 23:49:30.097804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.097682 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-maas-api-tls\") pod \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " Nov 03 23:49:30.097804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.097757 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-metrics-certs\") pod \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " Nov 03 23:49:30.097936 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.097897 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-kube-api-access-4h76t\" (UniqueName: \"kubernetes.io/projected/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-kube-api-access-4h76t\") pod \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\" (UID: \"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2\") " Nov 03 23:49:30.098408 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.098024 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-openshift-service-ca" pod "1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" (UID: "1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:49:30.098535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.098514 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:30.100239 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.100214 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-kube-api-access-4h76t" pod "1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" (UID: "1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2"). InnerVolumeSpecName "kube-api-access-4h76t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:49:30.100355 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.100329 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-metrics-certs" pod "1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" (UID: "1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:49:30.100355 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.100344 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-maas-api-tls" pod "1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" (UID: "1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:49:30.200068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.199969 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:30.200068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.200002 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:30.200068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.200014 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h76t\" (UniqueName: \"kubernetes.io/projected/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2-kube-api-access-4h76t\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:30.607534 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.607496 6208 generic.go:356] "Generic (PLEG): container finished" podID="1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" containerID="84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13" exitCode=0 Nov 03 23:49:30.607984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.607546 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" Nov 03 23:49:30.607984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.607574 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" event={"ID":"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2","Type":"ContainerDied","Data":"84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13"} Nov 03 23:49:30.607984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.607633 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt" event={"ID":"1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2","Type":"ContainerDied","Data":"fbe6e4fbf7ed7028fa6d38e8ade4382f14aa839d95b45cbee01865a2c5ee4f6e"} Nov 03 23:49:30.607984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.607656 6208 scope.go:122] "RemoveContainer" containerID="84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13" Nov 03 23:49:30.618975 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.618953 6208 scope.go:122] "RemoveContainer" containerID="84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13" Nov 03 23:49:30.619273 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:49:30.619251 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13\": container with ID starting with 84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13 not found: ID does not exist" containerID="84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13" Nov 03 23:49:30.619354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.619285 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13"} err="failed to get container status \"84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13\": rpc error: code = NotFound desc = could not find container \"84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13\": container with ID starting with 84bbc92ec2ae7e37913938179027540a7ee3931ef78c9bda0264bd7ad7438b13 not found: ID does not exist" Nov 03 23:49:30.637050 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.637009 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt"] Nov 03 23:49:30.640979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.640952 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a88e9c-84b9d5b79c-b75wt"] Nov 03 23:49:30.814857 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.814801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:30.929583 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:30.929485 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:31.295664 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:31.295628 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" path="/var/lib/kubelet/pods/1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2/volumes" Nov 03 23:49:31.760514 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:31.760482 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:32.754920 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:32.754884 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:33.559260 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:33.559213 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:34.633439 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:34.633398 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:35.592817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:35.592782 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:35.718429 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:49:35.718388 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:49:36.389404 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:36.389356 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:36.727192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:36.727157 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:36.852072 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:36.852027 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:37.279780 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:37.279751 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:37.518564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:37.518528 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:38.262336 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:38.262303 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:38.362772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:38.362742 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:38.465873 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:38.465833 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:38.880644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:38.880606 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:39.247836 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:39.247798 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:39.391794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:39.391740 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:40.644644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:40.644613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:41.121954 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.121918 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:41.592396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.592364 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t"] Nov 03 23:49:41.592976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.592963 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" containerName="maas-api" Nov 03 23:49:41.593025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.592977 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" containerName="maas-api" Nov 03 23:49:41.593025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.592991 6208 cpu_manager.go:414] "containerMap: removing container" podUID="41dac6ee-468a-4559-876e-f4ab60e85ba9" containerName="payload-pre-processing" Nov 03 23:49:41.593025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.592996 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41dac6ee-468a-4559-876e-f4ab60e85ba9" containerName="payload-pre-processing" Nov 03 23:49:41.593025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593012 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a" containerName="payload-processing" Nov 03 23:49:41.593025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593017 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a" containerName="payload-processing" Nov 03 23:49:41.593214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593027 6208 cpu_manager.go:414] "containerMap: removing container" podUID="6de4693f-7d97-4c22-b39c-517473ee9486" containerName="main" Nov 03 23:49:41.593214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593062 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de4693f-7d97-4c22-b39c-517473ee9486" containerName="main" Nov 03 23:49:41.593214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593086 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f4f71db5-a959-4b98-b7cf-6d2b6a34915f" containerName="payload-processing" Nov 03 23:49:41.593214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593090 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f71db5-a959-4b98-b7cf-6d2b6a34915f" containerName="payload-processing" Nov 03 23:49:41.593214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593192 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f71db5-a959-4b98-b7cf-6d2b6a34915f" containerName="payload-processing" Nov 03 23:49:41.593214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593206 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9c5e8d-abb5-4560-9aab-49c97e5ad9e2" containerName="maas-api" Nov 03 23:49:41.593214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593215 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="41dac6ee-468a-4559-876e-f4ab60e85ba9" containerName="payload-pre-processing" Nov 03 23:49:41.593423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593224 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bad7ff-6d9a-4fb4-9bd6-5f4be46d612a" containerName="payload-processing" Nov 03 23:49:41.593423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.593230 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de4693f-7d97-4c22-b39c-517473ee9486" containerName="main" Nov 03 23:49:41.598001 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.597978 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.607083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.607059 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c7e425-openshift-default-dockercfg-h6fns\"" Nov 03 23:49:41.616005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.615966 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t"] Nov 03 23:49:41.650185 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.650134 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:41.738057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.727916 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-istio-token\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.799684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.783017 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-envoy\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.799684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.783238 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bd2754-7d47-4cec-830e-45de493c7724-istiod-ca-cert\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.799684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.783492 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-credential-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.799684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.783673 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-data\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.799684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.783733 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-certs\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.799684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.783842 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.799684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.784072 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bd2754-7d47-4cec-830e-45de493c7724-istio-podinfo\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.799684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.784421 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5dz\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-kube-api-access-gt5dz\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.799684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.788949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:41.888542 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.888006 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-envoy\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.888875 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.888849 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bd2754-7d47-4cec-830e-45de493c7724-istiod-ca-cert\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.889243 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.889212 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-credential-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.889439 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.889420 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-data\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.889593 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.889554 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-certs\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.889736 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.889653 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.889736 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.889722 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bd2754-7d47-4cec-830e-45de493c7724-istio-podinfo\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.889853 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.889776 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5dz\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-kube-api-access-gt5dz\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.889908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.889880 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-istio-token\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.890153 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.890131 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-credential-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.890312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.890282 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bd2754-7d47-4cec-830e-45de493c7724-istiod-ca-cert\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.890528 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.890507 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-socket\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.890870 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.890847 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-data\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.891004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.890976 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-certs\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.893522 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.893501 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-envoy\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.895669 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.895645 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bd2754-7d47-4cec-830e-45de493c7724-istio-podinfo\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.906214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.904184 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-istio-token\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.906214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.904390 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5dz\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-kube-api-access-gt5dz\") pod \"e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.913953 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.913355 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:41.914718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:41.914664 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:42.128133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.128100 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:49:42.155759 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.155685 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:49:42.222250 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.222220 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t"] Nov 03 23:49:42.223001 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:49:42.222980 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bd2754_7d47_4cec_830e_45de493c7724.slice/crio-4ba716c1953d15749846f5cbecfdc6e551983054dbabefe632f3caf967f53c9f WatchSource:0}: Error finding container 4ba716c1953d15749846f5cbecfdc6e551983054dbabefe632f3caf967f53c9f: Status 404 returned error can't find the container with id 4ba716c1953d15749846f5cbecfdc6e551983054dbabefe632f3caf967f53c9f Nov 03 23:49:42.225862 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.225826 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:49:42.225939 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.225893 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:49:42.225939 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.225921 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:49:42.556816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.556780 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:42.846234 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.846152 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" event={"ID":"35bd2754-7d47-4cec-830e-45de493c7724","Type":"ContainerStarted","Data":"e3867151e611d3a1ca47618c771643f5390a2b39ce8ba97a2f78ebc3b7111e08"} Nov 03 23:49:42.846234 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.846206 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" event={"ID":"35bd2754-7d47-4cec-830e-45de493c7724","Type":"ContainerStarted","Data":"4ba716c1953d15749846f5cbecfdc6e551983054dbabefe632f3caf967f53c9f"} Nov 03 23:49:42.873194 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.873133 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" podStartSLOduration=1.873115543 podStartE2EDuration="1.873115543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:49:42.871249882 +0000 UTC m=+1201.693846044" watchObservedRunningTime="2026-11-03 23:49:42.873115543 +0000 UTC m=+1201.695711706" Nov 03 23:49:42.914179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:42.914152 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:43.455950 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:43.455917 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:43.919303 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:43.919229 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:44.253825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:44.253797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:44.353361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:44.353330 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:44.456288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:44.456253 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:44.861911 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:44.861874 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:44.862738 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:44.862720 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:49:44.964520 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:44.964484 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:45.091002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:45.090961 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:45.268654 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:45.268607 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:45.415942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:45.415893 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:46.062850 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:46.062813 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:46.352081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:46.351947 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:46.460517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:46.460472 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:47.188255 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:47.188225 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:48.262849 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:48.262799 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:49.195059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:49.189409 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:49.446195 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:49.446104 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:50.583799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:50.583768 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:50.781111 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:50.781075 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8"] Nov 03 23:49:50.791488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:50.791454 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p"] Nov 03 23:49:50.842983 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:50.842894 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:50.943562 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:50.943529 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:51.075746 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:51.075683 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:51.840228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:51.840193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:52.362844 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:52.362809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:52.459748 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:52.459706 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:52.688639 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:52.688503 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:52.859484 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:52.859447 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:53.224633 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:53.224584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:54.050370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:54.050331 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:54.159751 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:54.159703 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:54.944607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:54.944573 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:55.660440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.660411 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:55.774798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.774763 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:55.782449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.782412 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" podUID="20d84339-bf8a-4c2e-898c-4ba75c8f6df7" containerName="payload-processing" containerID="cri-o://1d48a33fc316e78ee42a17adcc2045fae913367ef8d77c09f228970192866ab8" gracePeriod=25 Nov 03 23:49:55.791984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.791922 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" podUID="ef9af4e8-9477-41ff-889c-f57421a80750" containerName="payload-pre-processing" containerID="cri-o://5a86e6add744d2a796467f83d289ae3d2bf7c88728e8a64e8eea2c6b8cdec3ae" gracePeriod=25 Nov 03 23:49:55.884932 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:49:55.884904 6208 logging.go:55] [core] [Channel #1578 SubChannel #1579]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.104:50052", ServerName: "10.133.0.104:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.104:50052: connect: connection refused" Nov 03 23:49:55.884932 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:49:55.884908 6208 logging.go:55] [core] [Channel #1577 SubChannel #1580]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.105:50052", ServerName: "10.133.0.105:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.105:50052: connect: connection refused" Nov 03 23:49:55.887606 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.887581 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:55.959510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.959479 6208 generic.go:356] "Generic (PLEG): container finished" podID="ef9af4e8-9477-41ff-889c-f57421a80750" containerID="5a86e6add744d2a796467f83d289ae3d2bf7c88728e8a64e8eea2c6b8cdec3ae" exitCode=0 Nov 03 23:49:55.959680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.959546 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" event={"ID":"ef9af4e8-9477-41ff-889c-f57421a80750","Type":"ContainerDied","Data":"5a86e6add744d2a796467f83d289ae3d2bf7c88728e8a64e8eea2c6b8cdec3ae"} Nov 03 23:49:55.961573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.961544 6208 generic.go:356] "Generic (PLEG): container finished" podID="20d84339-bf8a-4c2e-898c-4ba75c8f6df7" containerID="1d48a33fc316e78ee42a17adcc2045fae913367ef8d77c09f228970192866ab8" exitCode=0 Nov 03 23:49:55.962176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.961644 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" event={"ID":"20d84339-bf8a-4c2e-898c-4ba75c8f6df7","Type":"ContainerDied","Data":"1d48a33fc316e78ee42a17adcc2045fae913367ef8d77c09f228970192866ab8"} Nov 03 23:49:55.986484 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:55.983117 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:56.093116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.093089 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:49:56.119753 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.119728 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:49:56.156311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.156276 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-plugins-config-volume\") pod \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\" (UID: \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\") " Nov 03 23:49:56.156508 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.156402 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-kube-api-access-vjlsg\" (UniqueName: \"kubernetes.io/projected/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-kube-api-access-vjlsg\") pod \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\" (UID: \"20d84339-bf8a-4c2e-898c-4ba75c8f6df7\") " Nov 03 23:49:56.156753 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.156726 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-plugins-config-volume" pod "20d84339-bf8a-4c2e-898c-4ba75c8f6df7" (UID: "20d84339-bf8a-4c2e-898c-4ba75c8f6df7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:49:56.157229 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.157201 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:56.159771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.159743 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-kube-api-access-vjlsg" pod "20d84339-bf8a-4c2e-898c-4ba75c8f6df7" (UID: "20d84339-bf8a-4c2e-898c-4ba75c8f6df7"). InnerVolumeSpecName "kube-api-access-vjlsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:49:56.257959 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.257927 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9af4e8-9477-41ff-889c-f57421a80750-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9af4e8-9477-41ff-889c-f57421a80750-plugins-config-volume\") pod \"ef9af4e8-9477-41ff-889c-f57421a80750\" (UID: \"ef9af4e8-9477-41ff-889c-f57421a80750\") " Nov 03 23:49:56.258159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.258072 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9af4e8-9477-41ff-889c-f57421a80750-kube-api-access-4p8kj\" (UniqueName: \"kubernetes.io/projected/ef9af4e8-9477-41ff-889c-f57421a80750-kube-api-access-4p8kj\") pod \"ef9af4e8-9477-41ff-889c-f57421a80750\" (UID: \"ef9af4e8-9477-41ff-889c-f57421a80750\") " Nov 03 23:49:56.258328 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.258306 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9af4e8-9477-41ff-889c-f57421a80750-plugins-config-volume" pod "ef9af4e8-9477-41ff-889c-f57421a80750" (UID: "ef9af4e8-9477-41ff-889c-f57421a80750"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:49:56.258615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.258596 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef9af4e8-9477-41ff-889c-f57421a80750-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:56.258615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.258612 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjlsg\" (UniqueName: \"kubernetes.io/projected/20d84339-bf8a-4c2e-898c-4ba75c8f6df7-kube-api-access-vjlsg\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:56.260155 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.260132 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9af4e8-9477-41ff-889c-f57421a80750-kube-api-access-4p8kj" pod "ef9af4e8-9477-41ff-889c-f57421a80750" (UID: "ef9af4e8-9477-41ff-889c-f57421a80750"). InnerVolumeSpecName "kube-api-access-4p8kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:49:56.363417 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.363373 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p8kj\" (UniqueName: \"kubernetes.io/projected/ef9af4e8-9477-41ff-889c-f57421a80750-kube-api-access-4p8kj\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:49:56.385026 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.384992 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:56.686232 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.678469 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:56.884698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.884653 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" podUID="ef9af4e8-9477-41ff-889c-f57421a80750" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.104:50052\" within 1s: context deadline exceeded" Nov 03 23:49:56.884879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.884656 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" podUID="20d84339-bf8a-4c2e-898c-4ba75c8f6df7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.105:50052\" within 1s: context deadline exceeded" Nov 03 23:49:56.973707 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.973670 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" event={"ID":"ef9af4e8-9477-41ff-889c-f57421a80750","Type":"ContainerDied","Data":"aa61ad4bc5a6e709474df70fc69d0f8e497787e80dc9843416ee324612c48b2b"} Nov 03 23:49:56.973903 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.973738 6208 scope.go:122] "RemoveContainer" containerID="5a86e6add744d2a796467f83d289ae3d2bf7c88728e8a64e8eea2c6b8cdec3ae" Nov 03 23:49:56.973903 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.973681 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p" Nov 03 23:49:56.975985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.975953 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" event={"ID":"20d84339-bf8a-4c2e-898c-4ba75c8f6df7","Type":"ContainerDied","Data":"a32ef0a09fb9ce0157b638ec53092312df761b47ff56a04d4902f5186b7d7510"} Nov 03 23:49:56.976161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.976070 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8" Nov 03 23:49:56.987758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:56.987729 6208 scope.go:122] "RemoveContainer" containerID="1d48a33fc316e78ee42a17adcc2045fae913367ef8d77c09f228970192866ab8" Nov 03 23:49:57.005680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:57.005649 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p"] Nov 03 23:49:57.009508 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:57.009469 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88f34dbe-7f7b7bb887-qsj2p"] Nov 03 23:49:57.032819 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:57.032784 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8"] Nov 03 23:49:57.037362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:57.037324 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88f34dbe-757d598c96-r67s8"] Nov 03 23:49:57.295201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:57.295125 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d84339-bf8a-4c2e-898c-4ba75c8f6df7" path="/var/lib/kubelet/pods/20d84339-bf8a-4c2e-898c-4ba75c8f6df7/volumes" Nov 03 23:49:57.303262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:57.296307 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9af4e8-9477-41ff-889c-f57421a80750" path="/var/lib/kubelet/pods/ef9af4e8-9477-41ff-889c-f57421a80750/volumes" Nov 03 23:49:57.625700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:57.625590 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:58.473847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:58.473810 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:59.105569 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:59.105529 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:59.212584 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:59.212556 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:59.543265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:59.543225 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:49:59.713938 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:49:59.713902 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:00.093467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:00.093430 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm"] Nov 03 23:50:00.093756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:00.093727 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" containerID="cri-o://28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133" gracePeriod=30 Nov 03 23:50:00.132227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:00.132189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:00.987345 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:00.987306 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:01.087820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:01.087781 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:01.444285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:01.444252 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:01.544206 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:01.544172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:01.904838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:01.904797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:02.145679 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:02.145646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:03.040282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:03.040242 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:04.127602 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:04.127563 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:04.934077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:04.934024 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:05.053136 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.053097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:05.381236 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.381206 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:50:05.393595 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.393567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:05.505797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.505696 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-podinfo\") pod \"e009c7e2-89eb-42bc-bbfb-24decb053b12\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " Nov 03 23:50:05.505972 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.505808 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-kube-api-access-ptj2r\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-kube-api-access-ptj2r\") pod \"e009c7e2-89eb-42bc-bbfb-24decb053b12\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " Nov 03 23:50:05.505972 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.505845 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-certs\") pod \"e009c7e2-89eb-42bc-bbfb-24decb053b12\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " Nov 03 23:50:05.505972 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.505917 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-data\") pod \"e009c7e2-89eb-42bc-bbfb-24decb053b12\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " Nov 03 23:50:05.505972 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.505965 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-credential-socket\") pod \"e009c7e2-89eb-42bc-bbfb-24decb053b12\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " Nov 03 23:50:05.506247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.506058 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-envoy\") pod \"e009c7e2-89eb-42bc-bbfb-24decb053b12\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " Nov 03 23:50:05.506247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.506145 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e009c7e2-89eb-42bc-bbfb-24decb053b12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e009c7e2-89eb-42bc-bbfb-24decb053b12-istiod-ca-cert\") pod \"e009c7e2-89eb-42bc-bbfb-24decb053b12\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " Nov 03 23:50:05.506358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.506269 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-credential-socket" pod "e009c7e2-89eb-42bc-bbfb-24decb053b12" (UID: "e009c7e2-89eb-42bc-bbfb-24decb053b12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:05.506416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.506371 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-data" pod "e009c7e2-89eb-42bc-bbfb-24decb053b12" (UID: "e009c7e2-89eb-42bc-bbfb-24decb053b12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:05.506460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.506434 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-certs" pod "e009c7e2-89eb-42bc-bbfb-24decb053b12" (UID: "e009c7e2-89eb-42bc-bbfb-24decb053b12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:05.506510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.506485 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e009c7e2-89eb-42bc-bbfb-24decb053b12-istiod-ca-cert" pod "e009c7e2-89eb-42bc-bbfb-24decb053b12" (UID: "e009c7e2-89eb-42bc-bbfb-24decb053b12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:50:05.506855 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.506816 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-token\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-token\") pod \"e009c7e2-89eb-42bc-bbfb-24decb053b12\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " Nov 03 23:50:05.507209 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.506912 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-socket\") pod \"e009c7e2-89eb-42bc-bbfb-24decb053b12\" (UID: \"e009c7e2-89eb-42bc-bbfb-24decb053b12\") " Nov 03 23:50:05.507449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.507413 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-socket" pod "e009c7e2-89eb-42bc-bbfb-24decb053b12" (UID: "e009c7e2-89eb-42bc-bbfb-24decb053b12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:05.509165 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.509136 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:05.509274 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.509169 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:05.509274 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.509184 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:05.509274 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.509197 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:05.509274 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.509210 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e009c7e2-89eb-42bc-bbfb-24decb053b12-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:05.510326 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.510294 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-envoy" pod "e009c7e2-89eb-42bc-bbfb-24decb053b12" (UID: "e009c7e2-89eb-42bc-bbfb-24decb053b12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:05.510435 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.510344 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-kube-api-access-ptj2r" pod "e009c7e2-89eb-42bc-bbfb-24decb053b12" (UID: "e009c7e2-89eb-42bc-bbfb-24decb053b12"). InnerVolumeSpecName "kube-api-access-ptj2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:05.510503 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.510403 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-podinfo" pod "e009c7e2-89eb-42bc-bbfb-24decb053b12" (UID: "e009c7e2-89eb-42bc-bbfb-24decb053b12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:50:05.510503 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.510472 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-token" pod "e009c7e2-89eb-42bc-bbfb-24decb053b12" (UID: "e009c7e2-89eb-42bc-bbfb-24decb053b12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:05.611389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.611312 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:05.611389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.611353 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:05.611389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.611369 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e009c7e2-89eb-42bc-bbfb-24decb053b12-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:05.611389 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.611383 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptj2r\" (UniqueName: \"kubernetes.io/projected/e009c7e2-89eb-42bc-bbfb-24decb053b12-kube-api-access-ptj2r\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:05.704949 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:50:05.704910 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:50:05.910122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:05.909943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:06.046531 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.046497 6208 generic.go:356] "Generic (PLEG): container finished" podID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerID="28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133" exitCode=0 Nov 03 23:50:06.046719 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.046545 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" Nov 03 23:50:06.046719 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.046536 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" event={"ID":"e009c7e2-89eb-42bc-bbfb-24decb053b12","Type":"ContainerDied","Data":"28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133"} Nov 03 23:50:06.046719 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.046675 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm" event={"ID":"e009c7e2-89eb-42bc-bbfb-24decb053b12","Type":"ContainerDied","Data":"49b8f193d7fdca80142edc9bc4d7507c3f7754bf7c5d794a5f12ae4b4d9a07d6"} Nov 03 23:50:06.046719 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.046705 6208 scope.go:122] "RemoveContainer" containerID="28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133" Nov 03 23:50:06.060380 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.060355 6208 scope.go:122] "RemoveContainer" containerID="0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69" Nov 03 23:50:06.073171 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.073150 6208 scope.go:122] "RemoveContainer" containerID="28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133" Nov 03 23:50:06.073601 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:50:06.073552 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133\": container with ID starting with 28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133 not found: ID does not exist" containerID="28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133" Nov 03 23:50:06.073790 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.073593 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133"} err="failed to get container status \"28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133\": rpc error: code = NotFound desc = could not find container \"28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133\": container with ID starting with 28af34956c8b06383b3226742102fbd5bcc71e63b436b6d0174d58d78fe38133 not found: ID does not exist" Nov 03 23:50:06.073790 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.073619 6208 scope.go:122] "RemoveContainer" containerID="0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69" Nov 03 23:50:06.074975 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:50:06.073944 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69\": container with ID starting with 0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69 not found: ID does not exist" containerID="0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69" Nov 03 23:50:06.074975 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.074000 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69"} err="failed to get container status \"0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69\": rpc error: code = NotFound desc = could not find container \"0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69\": container with ID starting with 0f3f71e21bfee14447d93c09f12b5f764b709e5d22a9a1dab77d290835ba6a69 not found: ID does not exist" Nov 03 23:50:06.087426 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.087399 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm"] Nov 03 23:50:06.087518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.087454 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-88f34dbe-openshift-default-685d7945b5-q86nm"] Nov 03 23:50:06.129545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.129514 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:06.243751 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:06.243711 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:07.155418 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:07.155377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:07.296720 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:07.296686 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" path="/var/lib/kubelet/pods/e009c7e2-89eb-42bc-bbfb-24decb053b12/volumes" Nov 03 23:50:08.184087 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:08.184048 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:08.955825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:08.955783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:09.718898 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:09.718862 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z"] Nov 03 23:50:09.719366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:09.719142 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" podUID="6980ffec-ec71-4dd0-9037-d264003cbaac" containerName="maas-api" containerID="cri-o://9f790ae26feaa81e63116101adafea5bbdb6e8e82b55bd27cbd2199721cc05b9" gracePeriod=30 Nov 03 23:50:09.743722 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:09.740492 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:09.849605 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:09.849566 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:09.950357 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:09.950319 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:10.090810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.089402 6208 generic.go:356] "Generic (PLEG): container finished" podID="6980ffec-ec71-4dd0-9037-d264003cbaac" containerID="9f790ae26feaa81e63116101adafea5bbdb6e8e82b55bd27cbd2199721cc05b9" exitCode=0 Nov 03 23:50:10.090810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.089478 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" event={"ID":"6980ffec-ec71-4dd0-9037-d264003cbaac","Type":"ContainerDied","Data":"9f790ae26feaa81e63116101adafea5bbdb6e8e82b55bd27cbd2199721cc05b9"} Nov 03 23:50:10.090810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.089529 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" event={"ID":"6980ffec-ec71-4dd0-9037-d264003cbaac","Type":"ContainerDied","Data":"1d66213b0785f076d9d2a1aac948a7c05ea271a8ba245bfa77eee9ad567e268e"} Nov 03 23:50:10.090810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.089553 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d66213b0785f076d9d2a1aac948a7c05ea271a8ba245bfa77eee9ad567e268e" Nov 03 23:50:10.098057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.092259 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:50:10.170778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.170739 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6980ffec-ec71-4dd0-9037-d264003cbaac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6980ffec-ec71-4dd0-9037-d264003cbaac-openshift-service-ca\") pod \"6980ffec-ec71-4dd0-9037-d264003cbaac\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " Nov 03 23:50:10.173147 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.171655 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6980ffec-ec71-4dd0-9037-d264003cbaac-openshift-service-ca" pod "6980ffec-ec71-4dd0-9037-d264003cbaac" (UID: "6980ffec-ec71-4dd0-9037-d264003cbaac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:50:10.173147 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.172196 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6980ffec-ec71-4dd0-9037-d264003cbaac-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:10.273091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.273011 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6980ffec-ec71-4dd0-9037-d264003cbaac-kube-api-access-6fq72\" (UniqueName: \"kubernetes.io/projected/6980ffec-ec71-4dd0-9037-d264003cbaac-kube-api-access-6fq72\") pod \"6980ffec-ec71-4dd0-9037-d264003cbaac\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " Nov 03 23:50:10.273091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.273082 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-metrics-certs\") pod \"6980ffec-ec71-4dd0-9037-d264003cbaac\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " Nov 03 23:50:10.273394 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.273137 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-maas-api-tls\") pod \"6980ffec-ec71-4dd0-9037-d264003cbaac\" (UID: \"6980ffec-ec71-4dd0-9037-d264003cbaac\") " Nov 03 23:50:10.275136 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.275104 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6980ffec-ec71-4dd0-9037-d264003cbaac-kube-api-access-6fq72" pod "6980ffec-ec71-4dd0-9037-d264003cbaac" (UID: "6980ffec-ec71-4dd0-9037-d264003cbaac"). InnerVolumeSpecName "kube-api-access-6fq72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:10.275247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.275149 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-metrics-certs" pod "6980ffec-ec71-4dd0-9037-d264003cbaac" (UID: "6980ffec-ec71-4dd0-9037-d264003cbaac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:50:10.275247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.275239 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-maas-api-tls" pod "6980ffec-ec71-4dd0-9037-d264003cbaac" (UID: "6980ffec-ec71-4dd0-9037-d264003cbaac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:50:10.375190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.375152 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fq72\" (UniqueName: \"kubernetes.io/projected/6980ffec-ec71-4dd0-9037-d264003cbaac-kube-api-access-6fq72\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:10.375190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.375188 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:10.375190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.375198 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6980ffec-ec71-4dd0-9037-d264003cbaac-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:10.405750 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.405712 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:10.505719 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:10.505681 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:11.096606 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:11.096575 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z" Nov 03 23:50:11.122802 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:11.122770 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z"] Nov 03 23:50:11.126609 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:11.126583 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8cfc3c-5684d9df9d-jvd2z"] Nov 03 23:50:11.295981 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:11.295950 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6980ffec-ec71-4dd0-9037-d264003cbaac" path="/var/lib/kubelet/pods/6980ffec-ec71-4dd0-9037-d264003cbaac/volumes" Nov 03 23:50:11.435910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:11.435817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:11.655065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:11.655000 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:11.883965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:11.883926 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:12.096442 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.096407 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k"] Nov 03 23:50:12.097060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097026 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" Nov 03 23:50:12.097060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097057 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097078 6208 cpu_manager.go:414] "containerMap: removing container" podUID="ef9af4e8-9477-41ff-889c-f57421a80750" containerName="payload-pre-processing" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097085 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9af4e8-9477-41ff-889c-f57421a80750" containerName="payload-pre-processing" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097101 6208 cpu_manager.go:414] "containerMap: removing container" podUID="6980ffec-ec71-4dd0-9037-d264003cbaac" containerName="maas-api" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097106 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6980ffec-ec71-4dd0-9037-d264003cbaac" containerName="maas-api" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097132 6208 cpu_manager.go:414] "containerMap: removing container" podUID="20d84339-bf8a-4c2e-898c-4ba75c8f6df7" containerName="payload-processing" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097138 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d84339-bf8a-4c2e-898c-4ba75c8f6df7" containerName="payload-processing" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097243 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097255 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097263 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9af4e8-9477-41ff-889c-f57421a80750" containerName="payload-pre-processing" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097271 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="6980ffec-ec71-4dd0-9037-d264003cbaac" containerName="maas-api" Nov 03 23:50:12.097545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.097285 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d84339-bf8a-4c2e-898c-4ba75c8f6df7" containerName="payload-processing" Nov 03 23:50:12.101202 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.101180 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.107853 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.107828 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-d6e155-openshift-default-dockercfg-k7kb9\"" Nov 03 23:50:12.135094 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.111970 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k"] Nov 03 23:50:12.200868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.200834 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-socket\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.201062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.200882 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bcfb\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-kube-api-access-7bcfb\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.201062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.200999 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-envoy\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.201168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.201080 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad6532c-46b9-439c-88d3-395fe0b78a50-istiod-ca-cert\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.201168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.201104 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-credential-socket\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.201168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.201134 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-token\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.201332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.201307 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-data\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.201375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.201354 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-podinfo\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.201449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.201433 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-certs\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.277914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.272008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:12.310060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.306005 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad6532c-46b9-439c-88d3-395fe0b78a50-istiod-ca-cert\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.346480 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.313857 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad6532c-46b9-439c-88d3-395fe0b78a50-istiod-ca-cert\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.346670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.346424 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-credential-socket\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.346670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.346590 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-token\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.346792 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.346737 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-data\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.346792 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.346773 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-podinfo\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.346895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.346798 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-credential-socket\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.346895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.346829 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-certs\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.347007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.346937 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-socket\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.347007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.346985 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bcfb\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-kube-api-access-7bcfb\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.347131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.347052 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-data\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.347192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.347135 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-envoy\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.347510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.347488 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-certs\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.347748 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.347724 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-socket\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.350014 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.349985 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-envoy\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.375318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.361275 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-token\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.375318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.361694 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bcfb\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-kube-api-access-7bcfb\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.378969 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.378945 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-podinfo\") pod \"e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.418990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.418904 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:12.543779 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.543264 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:12.734357 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:50:12.734324 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad6532c_46b9_439c_88d3_395fe0b78a50.slice/crio-d26827c58d3a767f008beb7c1da93510a4890bb11448d5025db187e98698e39e WatchSource:0}: Error finding container d26827c58d3a767f008beb7c1da93510a4890bb11448d5025db187e98698e39e: Status 404 returned error can't find the container with id d26827c58d3a767f008beb7c1da93510a4890bb11448d5025db187e98698e39e Nov 03 23:50:12.735711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.735675 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k"] Nov 03 23:50:12.736971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.736938 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:12.737052 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.737006 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:12.737097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.737072 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:12.738882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:12.738856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:13.115809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:13.115728 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" event={"ID":"5ad6532c-46b9-439c-88d3-395fe0b78a50","Type":"ContainerStarted","Data":"8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf"} Nov 03 23:50:13.115809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:13.115779 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" event={"ID":"5ad6532c-46b9-439c-88d3-395fe0b78a50","Type":"ContainerStarted","Data":"d26827c58d3a767f008beb7c1da93510a4890bb11448d5025db187e98698e39e"} Nov 03 23:50:13.145926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:13.145872 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podStartSLOduration=1.145855638 podStartE2EDuration="1.145855638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:13.145351434 +0000 UTC m=+1231.967947632" watchObservedRunningTime="2026-11-03 23:50:13.145855638 +0000 UTC m=+1231.968451801" Nov 03 23:50:13.421322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:13.419532 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:13.427358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:13.427328 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:13.469584 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:13.469555 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:13.569619 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:13.569584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:13.670173 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:13.670136 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:14.045672 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:14.045633 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz"] Nov 03 23:50:14.050767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:14.050726 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a5ff3c8-85vxz"] Nov 03 23:50:14.123679 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:14.123639 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:14.124872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:14.124846 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:50:14.407778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:14.407688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:15.296002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:15.295965 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="071f7045-14b1-45ae-a81e-ab4de2ee93c5" path="/var/lib/kubelet/pods/071f7045-14b1-45ae-a81e-ab4de2ee93c5/volumes" Nov 03 23:50:15.340695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:15.340663 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:15.624117 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:15.623947 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:15.734314 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:15.734268 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:15.838495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:15.838456 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:16.211313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.211278 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:16.339629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.339594 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:16.441659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.441624 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:16.542679 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.542595 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:16.677325 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.677290 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:16.715168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.715138 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9"] Nov 03 23:50:16.715732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.715719 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" Nov 03 23:50:16.715786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.715733 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e009c7e2-89eb-42bc-bbfb-24decb053b12" containerName="istio-proxy" Nov 03 23:50:16.720671 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.720650 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:16.734416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.734391 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d6e155\"" Nov 03 23:50:16.737364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.737341 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9"] Nov 03 23:50:16.742152 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.742133 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8"] Nov 03 23:50:16.746364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.746342 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:16.755166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.755132 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d6e155-dockercfg-bk8dr\"" Nov 03 23:50:16.779478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.779447 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8"] Nov 03 23:50:16.788575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.788548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:16.840181 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.840096 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rfs\" (UniqueName: \"kubernetes.io/projected/28e366c5-9003-4bce-bea7-5c131ada5520-kube-api-access-c9rfs\") pod \"payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8\" (UID: \"28e366c5-9003-4bce-bea7-5c131ada5520\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:16.840319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.840234 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e366c5-9003-4bce-bea7-5c131ada5520-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8\" (UID: \"28e366c5-9003-4bce-bea7-5c131ada5520\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:16.840404 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.840382 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fr58\" (UniqueName: \"kubernetes.io/projected/f004907f-237a-4f81-aa9d-e8d28ddb76cf-kube-api-access-9fr58\") pod \"payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9\" (UID: \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:16.840497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.840483 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f004907f-237a-4f81-aa9d-e8d28ddb76cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9\" (UID: \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:16.946455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.946411 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rfs\" (UniqueName: \"kubernetes.io/projected/28e366c5-9003-4bce-bea7-5c131ada5520-kube-api-access-c9rfs\") pod \"payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8\" (UID: \"28e366c5-9003-4bce-bea7-5c131ada5520\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:16.946889 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.946861 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e366c5-9003-4bce-bea7-5c131ada5520-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8\" (UID: \"28e366c5-9003-4bce-bea7-5c131ada5520\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:16.947013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.947000 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fr58\" (UniqueName: \"kubernetes.io/projected/f004907f-237a-4f81-aa9d-e8d28ddb76cf-kube-api-access-9fr58\") pod \"payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9\" (UID: \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:16.947128 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.947088 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f004907f-237a-4f81-aa9d-e8d28ddb76cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9\" (UID: \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:16.947989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.947959 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e366c5-9003-4bce-bea7-5c131ada5520-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8\" (UID: \"28e366c5-9003-4bce-bea7-5c131ada5520\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:16.949263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.949238 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f004907f-237a-4f81-aa9d-e8d28ddb76cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9\" (UID: \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:16.969388 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.969355 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rfs\" (UniqueName: \"kubernetes.io/projected/28e366c5-9003-4bce-bea7-5c131ada5520-kube-api-access-c9rfs\") pod \"payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8\" (UID: \"28e366c5-9003-4bce-bea7-5c131ada5520\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:16.977271 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:16.977189 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fr58\" (UniqueName: \"kubernetes.io/projected/f004907f-237a-4f81-aa9d-e8d28ddb76cf-kube-api-access-9fr58\") pod \"payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9\" (UID: \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:17.036657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:17.036610 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:17.078084 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:17.078050 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:17.335543 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:17.335515 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9"] Nov 03 23:50:17.335824 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:50:17.335800 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf004907f_237a_4f81_aa9d_e8d28ddb76cf.slice/crio-a656d6f3bd27679672f75ccb8749d2ae224915ddc49f5f7c8a6d6d4e9f8a3fb5 WatchSource:0}: Error finding container a656d6f3bd27679672f75ccb8749d2ae224915ddc49f5f7c8a6d6d4e9f8a3fb5: Status 404 returned error can't find the container with id a656d6f3bd27679672f75ccb8749d2ae224915ddc49f5f7c8a6d6d4e9f8a3fb5 Nov 03 23:50:17.405761 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:50:17.405731 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e366c5_9003_4bce_bea7_5c131ada5520.slice/crio-4236dbce6cdb2682d901619f2439c82e37ee1db1af95f6d6f70920c5184d9dd0 WatchSource:0}: Error finding container 4236dbce6cdb2682d901619f2439c82e37ee1db1af95f6d6f70920c5184d9dd0: Status 404 returned error can't find the container with id 4236dbce6cdb2682d901619f2439c82e37ee1db1af95f6d6f70920c5184d9dd0 Nov 03 23:50:17.406384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:17.406351 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8"] Nov 03 23:50:17.463355 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:17.463226 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:18.169204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:18.169169 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" event={"ID":"28e366c5-9003-4bce-bea7-5c131ada5520","Type":"ContainerStarted","Data":"101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c"} Nov 03 23:50:18.169397 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:18.169231 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" event={"ID":"28e366c5-9003-4bce-bea7-5c131ada5520","Type":"ContainerStarted","Data":"4236dbce6cdb2682d901619f2439c82e37ee1db1af95f6d6f70920c5184d9dd0"} Nov 03 23:50:18.169397 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:18.169378 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:18.171024 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:18.170996 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" event={"ID":"f004907f-237a-4f81-aa9d-e8d28ddb76cf","Type":"ContainerStarted","Data":"443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e"} Nov 03 23:50:18.171143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:18.171054 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" event={"ID":"f004907f-237a-4f81-aa9d-e8d28ddb76cf","Type":"ContainerStarted","Data":"a656d6f3bd27679672f75ccb8749d2ae224915ddc49f5f7c8a6d6d4e9f8a3fb5"} Nov 03 23:50:18.171143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:18.171085 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:18.190337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:18.190289 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" podStartSLOduration=2.190277118 podStartE2EDuration="2.190277118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:18.188259667 +0000 UTC m=+1237.010855830" watchObservedRunningTime="2026-11-03 23:50:18.190277118 +0000 UTC m=+1237.012873280" Nov 03 23:50:18.208689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:18.208647 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" podStartSLOduration=2.208636294 podStartE2EDuration="2.208636294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:18.206361904 +0000 UTC m=+1237.028958120" watchObservedRunningTime="2026-11-03 23:50:18.208636294 +0000 UTC m=+1237.031232455" Nov 03 23:50:18.302774 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:18.302742 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:19.122403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:19.122360 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:19.412063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:19.411937 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:19.516204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:19.516169 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:19.905549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:19.905517 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:20.067096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:20.067062 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp"] Nov 03 23:50:20.067274 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:20.067255 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" containerID="cri-o://c4c957e5968d8146c5259bb227b383825bcd35f340adf73c27bcbd11d5dd9799" gracePeriod=30 Nov 03 23:50:20.325009 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:20.324974 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:20.445320 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:20.445286 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:20.563058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:20.556081 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:21.032595 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.032559 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9"] Nov 03 23:50:21.036176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.036151 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24b4fa05-g5hs9"] Nov 03 23:50:21.299257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.299174 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e70338-2c45-448d-bacd-e6e8ff5409c7" path="/var/lib/kubelet/pods/e2e70338-2c45-448d-bacd-e6e8ff5409c7/volumes" Nov 03 23:50:21.455212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.455179 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:21.523007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.522971 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6"] Nov 03 23:50:21.528556 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.528534 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:21.533772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.533733 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c20f75-dockercfg-hctrs\"" Nov 03 23:50:21.537860 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.537836 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6"] Nov 03 23:50:21.557141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.557061 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:21.603023 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.602992 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c20f75\"" Nov 03 23:50:21.635063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.625783 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrtn\" (UniqueName: \"kubernetes.io/projected/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-kube-api-access-5vrtn\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6\" (UID: \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:21.635063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.625977 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6\" (UID: \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:21.730145 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.729088 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrtn\" (UniqueName: \"kubernetes.io/projected/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-kube-api-access-5vrtn\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6\" (UID: \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:21.730145 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.729281 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6\" (UID: \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:21.730444 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.730404 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6\" (UID: \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:21.755083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.755016 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrtn\" (UniqueName: \"kubernetes.io/projected/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-kube-api-access-5vrtn\") pod \"payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6\" (UID: \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:21.848101 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.847985 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:21.854934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.854900 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:21.958690 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:21.955578 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:22.148459 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.148427 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6"] Nov 03 23:50:22.149816 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:50:22.149788 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1f9716_dbbb_4ffb_a4c7_95844b82909c.slice/crio-3386d6ddf1923b9471900bf62d3c8c16cce4eda4389c4c9238b48e048c41e150 WatchSource:0}: Error finding container 3386d6ddf1923b9471900bf62d3c8c16cce4eda4389c4c9238b48e048c41e150: Status 404 returned error can't find the container with id 3386d6ddf1923b9471900bf62d3c8c16cce4eda4389c4c9238b48e048c41e150 Nov 03 23:50:22.212984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.212951 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" event={"ID":"3f1f9716-dbbb-4ffb-a4c7-95844b82909c","Type":"ContainerStarted","Data":"3386d6ddf1923b9471900bf62d3c8c16cce4eda4389c4c9238b48e048c41e150"} Nov 03 23:50:22.320364 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.317230 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l"] Nov 03 23:50:22.322571 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.322546 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.327316 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.327229 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-695f03-openshift-default-dockercfg-8p95k\"" Nov 03 23:50:22.338318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.338290 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l"] Nov 03 23:50:22.342106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.342071 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm58c\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-kube-api-access-rm58c\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.342222 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.342201 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-data\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.342306 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.342246 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3131912f-efdb-4051-8ef0-380a18b10bc1-istiod-ca-cert\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.342306 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.342271 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-socket\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.342461 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.342441 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-certs\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.342521 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.342494 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-envoy\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.342695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.342672 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-credential-socket\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.343185 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.343162 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-podinfo\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.343283 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.343203 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-token\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.367936 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.367883 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:22.453738 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.453627 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm58c\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-kube-api-access-rm58c\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.453913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.453737 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-data\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.453913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.453775 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3131912f-efdb-4051-8ef0-380a18b10bc1-istiod-ca-cert\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.453913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.453805 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-socket\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.454085 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.453911 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-certs\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.454085 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.453937 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-envoy\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.466537 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.466494 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-credential-socket\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.467977 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.467139 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-data\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.467977 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.467806 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3131912f-efdb-4051-8ef0-380a18b10bc1-istiod-ca-cert\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.467977 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.454027 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-credential-socket\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.468235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.467978 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-podinfo\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.468235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.468012 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-token\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.468235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.468079 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-socket\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.468559 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.468519 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-certs\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.471484 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.471461 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-envoy\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.475225 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.474393 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-podinfo\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.507644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.507559 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:22.510188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.510126 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm58c\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-kube-api-access-rm58c\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.550053 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.548630 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-token\") pod \"e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.640387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.640354 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:22.940288 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:50:22.940181 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3131912f_efdb_4051_8ef0_380a18b10bc1.slice/crio-883ad02bc01d4167c767acf75fff637b3e4f017747d0a8ee6830306aa6df5eff WatchSource:0}: Error finding container 883ad02bc01d4167c767acf75fff637b3e4f017747d0a8ee6830306aa6df5eff: Status 404 returned error can't find the container with id 883ad02bc01d4167c767acf75fff637b3e4f017747d0a8ee6830306aa6df5eff Nov 03 23:50:22.942878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.942831 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:22.943006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.942931 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:22.943006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.942978 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:22.945397 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:22.945373 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l"] Nov 03 23:50:23.222673 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:23.222583 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" event={"ID":"3f1f9716-dbbb-4ffb-a4c7-95844b82909c","Type":"ContainerStarted","Data":"dedb5441fb451fb921ffcf416d2117814fc30a7c8ab101aba8eb1dad35c9cf97"} Nov 03 23:50:23.222835 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:23.222803 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:23.224439 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:23.224412 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" event={"ID":"3131912f-efdb-4051-8ef0-380a18b10bc1","Type":"ContainerStarted","Data":"1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7"} Nov 03 23:50:23.224553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:23.224462 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" event={"ID":"3131912f-efdb-4051-8ef0-380a18b10bc1","Type":"ContainerStarted","Data":"883ad02bc01d4167c767acf75fff637b3e4f017747d0a8ee6830306aa6df5eff"} Nov 03 23:50:23.228134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:23.228112 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:23.249588 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:23.249538 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" podStartSLOduration=2.249522057 podStartE2EDuration="2.249522057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:23.246152251 +0000 UTC m=+1242.068748423" watchObservedRunningTime="2026-11-03 23:50:23.249522057 +0000 UTC m=+1242.072118220" Nov 03 23:50:23.276219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:23.276167 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" podStartSLOduration=1.276151242 podStartE2EDuration="1.276151242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:23.272051166 +0000 UTC m=+1242.094647335" watchObservedRunningTime="2026-11-03 23:50:23.276151242 +0000 UTC m=+1242.098747406" Nov 03 23:50:23.641338 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:23.641256 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:24.038158 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:24.038117 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:24.647199 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:24.647167 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:25.105547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.105511 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj"] Nov 03 23:50:25.110375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.110346 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.119082 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.119051 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-d6e155-serving-cert\"" Nov 03 23:50:25.126625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.126589 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj"] Nov 03 23:50:25.168048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.168010 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:25.236191 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.236149 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-metrics-certs\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.236379 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.236246 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdvr\" (UniqueName: \"kubernetes.io/projected/5b605151-325f-4bd0-b11d-23a2225bc799-kube-api-access-9pdvr\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.236379 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.236307 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b605151-325f-4bd0-b11d-23a2225bc799-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.236379 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.236351 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.254403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.253246 6208 generic.go:356] "Generic (PLEG): container finished" podID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerID="c4c957e5968d8146c5259bb227b383825bcd35f340adf73c27bcbd11d5dd9799" exitCode=0 Nov 03 23:50:25.254403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.253333 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" event={"ID":"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7","Type":"ContainerDied","Data":"c4c957e5968d8146c5259bb227b383825bcd35f340adf73c27bcbd11d5dd9799"} Nov 03 23:50:25.256110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.256086 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:25.257098 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.257028 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:50:25.270564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.270532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:25.359449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.359359 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-metrics-certs\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.359621 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.359473 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdvr\" (UniqueName: \"kubernetes.io/projected/5b605151-325f-4bd0-b11d-23a2225bc799-kube-api-access-9pdvr\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.360419 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.360394 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b605151-325f-4bd0-b11d-23a2225bc799-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.361141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.361119 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.361285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.361019 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b605151-325f-4bd0-b11d-23a2225bc799-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.364066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.364009 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-maas-api-tls\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.369337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.369318 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-metrics-certs\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.372925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.372893 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdvr\" (UniqueName: \"kubernetes.io/projected/5b605151-325f-4bd0-b11d-23a2225bc799-kube-api-access-9pdvr\") pod \"maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.375120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.375095 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:25.410315 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.410278 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:50:25.468700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.468670 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:25.563342 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563305 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-token\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-token\") pod \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " Nov 03 23:50:25.563517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563388 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-certs\") pod \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " Nov 03 23:50:25.563517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563471 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-credential-socket\") pod \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " Nov 03 23:50:25.563645 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563620 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-data\") pod \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " Nov 03 23:50:25.563709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563655 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-kube-api-access-hjnx6\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-kube-api-access-hjnx6\") pod \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " Nov 03 23:50:25.563709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563691 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-podinfo\") pod \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " Nov 03 23:50:25.563810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563733 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-certs" pod "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" (UID: "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:25.563810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563753 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istiod-ca-cert\") pod \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " Nov 03 23:50:25.563810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563744 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-credential-socket" pod "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" (UID: "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:25.563965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563805 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-envoy\") pod \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " Nov 03 23:50:25.563965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.563854 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-socket\") pod \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\" (UID: \"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7\") " Nov 03 23:50:25.564079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.564012 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-data" pod "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" (UID: "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:25.564557 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.564318 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-socket" pod "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" (UID: "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:25.564716 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.564682 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istiod-ca-cert" pod "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" (UID: "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:50:25.565018 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.564995 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:25.565166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.565024 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:25.565166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.565056 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:25.565166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.565069 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:25.565166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.565085 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:25.566907 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.566882 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-token" pod "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" (UID: "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:25.566994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.566908 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-envoy" pod "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" (UID: "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:25.568264 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.568235 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-kube-api-access-hjnx6" pod "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" (UID: "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7"). InnerVolumeSpecName "kube-api-access-hjnx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:25.568767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.568744 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-podinfo" pod "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" (UID: "65ddc9ee-841e-4f8c-a4b8-8343b4c470f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:50:25.666721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.666627 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjnx6\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-kube-api-access-hjnx6\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:25.666721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.666668 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:25.666721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.666683 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:25.666721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.666696 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:25.705361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.705279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:25.766571 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.766544 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj"] Nov 03 23:50:25.766687 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:50:25.766638 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b605151_325f_4bd0_b11d_23a2225bc799.slice/crio-1d216de375d94906be0355d55126c94cbd90b55dba67c93077fde47982b6a2fe WatchSource:0}: Error finding container 1d216de375d94906be0355d55126c94cbd90b55dba67c93077fde47982b6a2fe: Status 404 returned error can't find the container with id 1d216de375d94906be0355d55126c94cbd90b55dba67c93077fde47982b6a2fe Nov 03 23:50:25.768179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.768161 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:50:25.844910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:25.844882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:26.297560 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:26.297517 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" event={"ID":"5b605151-325f-4bd0-b11d-23a2225bc799","Type":"ContainerStarted","Data":"1d216de375d94906be0355d55126c94cbd90b55dba67c93077fde47982b6a2fe"} Nov 03 23:50:26.299382 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:26.299355 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" Nov 03 23:50:26.299497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:26.299373 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp" event={"ID":"65ddc9ee-841e-4f8c-a4b8-8343b4c470f7","Type":"ContainerDied","Data":"711bf00d8b4f68fd5116a27a39202e67adbdb355009941721ca903ea2b9fa622"} Nov 03 23:50:26.299497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:26.299442 6208 scope.go:122] "RemoveContainer" containerID="c4c957e5968d8146c5259bb227b383825bcd35f340adf73c27bcbd11d5dd9799" Nov 03 23:50:26.334384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:26.334312 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp"] Nov 03 23:50:26.338971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:26.338927 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xwvtp"] Nov 03 23:50:26.935724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:26.935691 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:27.082245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:27.076935 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:27.206617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:27.206583 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:27.350251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:27.350214 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" path="/var/lib/kubelet/pods/65ddc9ee-841e-4f8c-a4b8-8343b4c470f7/volumes" Nov 03 23:50:27.356213 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:27.356178 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" event={"ID":"5b605151-325f-4bd0-b11d-23a2225bc799","Type":"ContainerStarted","Data":"615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48"} Nov 03 23:50:27.356448 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:27.356428 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:27.385021 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:27.384970 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" podStartSLOduration=1.772857028 podStartE2EDuration="2.384954927s" totalImagesPullingTime="612.097899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:27.380384552 +0000 UTC m=+1246.202980713" watchObservedRunningTime="2026-11-03 23:50:27.384954927 +0000 UTC m=+1246.207551104" Nov 03 23:50:28.047698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:28.047664 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:29.201197 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:29.201166 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:50:29.241276 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:29.241228 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:50:29.274058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:29.269173 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:30.051093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:30.051020 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:30.151363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:30.151328 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:30.735459 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:30.735423 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:30.950855 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:30.950813 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:31.280755 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:31.280723 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:31.560111 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:31.558236 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:32.027719 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:32.027687 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:32.125212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:32.125147 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:32.312630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:32.312541 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:33.062058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:33.061122 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:34.086710 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:34.086670 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:34.236077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:34.235592 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:34.520566 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:34.520537 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:50:35.392635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:35.392600 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:35.711235 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:50:35.711185 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:50:35.750376 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:35.750331 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:35.849781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:35.849751 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:36.343862 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:36.343823 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:36.982882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:36.982846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:37.104147 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:37.104087 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:37.443923 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:37.443815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:38.234669 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:38.234636 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:38.336413 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:38.336378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:38.433566 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:38.433534 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:39.238548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:39.238501 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:40.125197 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:40.125156 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:40.677873 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:40.677839 6208 generic.go:356] "Generic (PLEG): container finished" podID="35bd2754-7d47-4cec-830e-45de493c7724" containerID="e3867151e611d3a1ca47618c771643f5390a2b39ce8ba97a2f78ebc3b7111e08" exitCode=0 Nov 03 23:50:40.678409 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:40.677920 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" event={"ID":"35bd2754-7d47-4cec-830e-45de493c7724","Type":"ContainerDied","Data":"e3867151e611d3a1ca47618c771643f5390a2b39ce8ba97a2f78ebc3b7111e08"} Nov 03 23:50:40.678491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:40.678445 6208 scope.go:122] "RemoveContainer" containerID="e3867151e611d3a1ca47618c771643f5390a2b39ce8ba97a2f78ebc3b7111e08" Nov 03 23:50:40.680045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:40.680000 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:40.680242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:40.680222 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:40.680304 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:40.680264 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:40.698589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:40.698567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:41.239224 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:41.239190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:41.702246 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:41.702122 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" event={"ID":"35bd2754-7d47-4cec-830e-45de493c7724","Type":"ContainerStarted","Data":"a599e8dc7418d9756f8081eea1f2c283b8901e4139f70eb77cf962bb9d7cd5dc"} Nov 03 23:50:41.914162 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:41.914124 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:50:41.915768 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:41.915732 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 23:50:41.915888 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:41.915780 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 23:50:42.179715 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:42.179678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:42.914226 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:42.914188 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 23:50:42.914596 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:42.914257 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 23:50:42.946651 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:42.943993 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:43.802917 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:43.802745 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:43.906096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:43.906060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:43.914170 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:43.914140 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 23:50:43.914367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:43.914347 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 23:50:44.530087 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:44.530019 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:44.874121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:44.873118 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:50:44.914762 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:44.914725 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 23:50:44.915196 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:44.914783 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 23:50:45.588582 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:45.588548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:45.914587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:45.914109 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 03 23:50:45.914587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:45.914160 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 03 23:50:46.481121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:46.481081 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp"] Nov 03 23:50:46.481956 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:46.481919 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" containerID="cri-o://9688b0138abe642150a9c15c017e6400e40f18627f37dad785c4f97079e4ace7" gracePeriod=30 Nov 03 23:50:46.487149 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:46.487124 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl"] Nov 03 23:50:46.487368 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:46.487342 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" containerID="cri-o://c974345fc6e21bf1dbbb62bc345fc24522bd0305f7478255cafd2832facb89cc" gracePeriod=30 Nov 03 23:50:46.698974 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:46.698928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:46.944416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:46.944348 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:50:46.946317 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:46.946292 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:50:47.609988 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.609952 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s"] Nov 03 23:50:47.611098 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.611074 6208 cpu_manager.go:414] "containerMap: removing container" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" Nov 03 23:50:47.611098 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.611098 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" Nov 03 23:50:47.611319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.611304 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ddc9ee-841e-4f8c-a4b8-8343b4c470f7" containerName="istio-proxy" Nov 03 23:50:47.615459 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.615437 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.629065 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.629019 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c83d52-serving-cert\"" Nov 03 23:50:47.633818 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.633793 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s"] Nov 03 23:50:47.662311 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.662276 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:47.710413 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.710380 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.710595 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.710498 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-maas-api-tls\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.710595 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.710556 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv845\" (UniqueName: \"kubernetes.io/projected/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-kube-api-access-sv845\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.710838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.710816 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-metrics-certs\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.762132 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.762086 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:47.812834 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.812800 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-maas-api-tls\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.813256 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.813213 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv845\" (UniqueName: \"kubernetes.io/projected/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-kube-api-access-sv845\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.813704 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.813436 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-metrics-certs\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.813704 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.813619 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.814371 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.814351 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.816160 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.816137 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-maas-api-tls\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.819577 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.819548 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-metrics-certs\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.825771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.825741 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv845\" (UniqueName: \"kubernetes.io/projected/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-kube-api-access-sv845\") pod \"maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:47.940697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:47.940607 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:48.266609 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:48.263225 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s"] Nov 03 23:50:48.555939 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:48.555858 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:48.780863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:48.780823 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" event={"ID":"0c7889d7-d920-4d3a-bd33-8ea7e07041a0","Type":"ContainerStarted","Data":"42236781cf460c3cf9b7c9cce6d7dd06c2d016b28b07e0ec0303667d95b8e4bb"} Nov 03 23:50:49.323425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:49.323395 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:49.789947 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:49.789910 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" event={"ID":"0c7889d7-d920-4d3a-bd33-8ea7e07041a0","Type":"ContainerStarted","Data":"50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c"} Nov 03 23:50:49.790367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:49.789973 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:49.811841 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:49.811783 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" podStartSLOduration=2.295393213 podStartE2EDuration="2.811769087s" totalImagesPullingTime="516.375874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:49.810346 +0000 UTC m=+1268.632942159" watchObservedRunningTime="2026-11-03 23:50:49.811769087 +0000 UTC m=+1268.634365248" Nov 03 23:50:50.230142 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:50.230095 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:50.592231 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:50.592124 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6"] Nov 03 23:50:50.667949 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:50.667911 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:50.762951 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:50.762918 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:50.932781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:50.932670 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:51.338684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.338647 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:51.822936 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.822893 6208 generic.go:356] "Generic (PLEG): container finished" podID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerID="c974345fc6e21bf1dbbb62bc345fc24522bd0305f7478255cafd2832facb89cc" exitCode=0 Nov 03 23:50:51.823124 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.822990 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" event={"ID":"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642","Type":"ContainerDied","Data":"c974345fc6e21bf1dbbb62bc345fc24522bd0305f7478255cafd2832facb89cc"} Nov 03 23:50:51.823124 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.823084 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" event={"ID":"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642","Type":"ContainerDied","Data":"ee9012c4f70e8eaf4eebc8564ccc2076c20d8472c19389ab313bfc2037822cab"} Nov 03 23:50:51.823124 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.823115 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9012c4f70e8eaf4eebc8564ccc2076c20d8472c19389ab313bfc2037822cab" Nov 03 23:50:51.825519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.825488 6208 generic.go:356] "Generic (PLEG): container finished" podID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerID="9688b0138abe642150a9c15c017e6400e40f18627f37dad785c4f97079e4ace7" exitCode=0 Nov 03 23:50:51.825648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.825626 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" event={"ID":"7bce0247-4a53-4c8b-9c84-152d2b268905","Type":"ContainerDied","Data":"9688b0138abe642150a9c15c017e6400e40f18627f37dad785c4f97079e4ace7"} Nov 03 23:50:51.825729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.825678 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" event={"ID":"7bce0247-4a53-4c8b-9c84-152d2b268905","Type":"ContainerDied","Data":"ca8af744dd6a77d78439c7162a86faa4506ceaf8d0c66d1f9c6181967e8bee0b"} Nov 03 23:50:51.825729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.825704 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca8af744dd6a77d78439c7162a86faa4506ceaf8d0c66d1f9c6181967e8bee0b" Nov 03 23:50:51.826506 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.826257 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:50:51.836885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.836832 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965101 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-data\") pod \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965178 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-socket\") pod \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965246 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-credential-socket\") pod \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965279 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-credential-socket\") pod \"7bce0247-4a53-4c8b-9c84-152d2b268905\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965307 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-certs\") pod \"7bce0247-4a53-4c8b-9c84-152d2b268905\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965338 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-kube-api-access-sxdrd\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-kube-api-access-sxdrd\") pod \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965387 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-token\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-token\") pod \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965418 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-podinfo\") pod \"7bce0247-4a53-4c8b-9c84-152d2b268905\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965525 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-kube-api-access-g4wgq\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-kube-api-access-g4wgq\") pod \"7bce0247-4a53-4c8b-9c84-152d2b268905\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965624 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bce0247-4a53-4c8b-9c84-152d2b268905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bce0247-4a53-4c8b-9c84-152d2b268905-istiod-ca-cert\") pod \"7bce0247-4a53-4c8b-9c84-152d2b268905\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965678 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-socket\") pod \"7bce0247-4a53-4c8b-9c84-152d2b268905\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965786 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-podinfo\") pod \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965813 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-envoy\") pod \"7bce0247-4a53-4c8b-9c84-152d2b268905\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965843 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-token\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-token\") pod \"7bce0247-4a53-4c8b-9c84-152d2b268905\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965874 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-data\") pod \"7bce0247-4a53-4c8b-9c84-152d2b268905\" (UID: \"7bce0247-4a53-4c8b-9c84-152d2b268905\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965915 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istiod-ca-cert\") pod \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.965957 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-certs\") pod \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " Nov 03 23:50:51.967204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.966012 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-envoy\") pod \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\" (UID: \"0fba9ea6-4d1f-46e6-bf0f-6dcafe283642\") " Nov 03 23:50:51.991580 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.991532 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istiod-ca-cert" pod "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" (UID: "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:50:51.991852 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.991823 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-data" pod "7bce0247-4a53-4c8b-9c84-152d2b268905" (UID: "7bce0247-4a53-4c8b-9c84-152d2b268905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.007066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:51.993806 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-credential-socket" pod "7bce0247-4a53-4c8b-9c84-152d2b268905" (UID: "7bce0247-4a53-4c8b-9c84-152d2b268905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.007066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.001223 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-token" pod "7bce0247-4a53-4c8b-9c84-152d2b268905" (UID: "7bce0247-4a53-4c8b-9c84-152d2b268905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:52.007066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.004592 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-certs" pod "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" (UID: "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.007066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.005023 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bce0247-4a53-4c8b-9c84-152d2b268905-istiod-ca-cert" pod "7bce0247-4a53-4c8b-9c84-152d2b268905" (UID: "7bce0247-4a53-4c8b-9c84-152d2b268905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:50:52.007066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.005407 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-certs" pod "7bce0247-4a53-4c8b-9c84-152d2b268905" (UID: "7bce0247-4a53-4c8b-9c84-152d2b268905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.007066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.006791 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-credential-socket" pod "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" (UID: "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.007066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.006844 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-socket" pod "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" (UID: "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.007481 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.007451 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-data" pod "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" (UID: "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.007957 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.007930 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-kube-api-access-sxdrd" pod "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" (UID: "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642"). InnerVolumeSpecName "kube-api-access-sxdrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:52.015478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.013220 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-socket" pod "7bce0247-4a53-4c8b-9c84-152d2b268905" (UID: "7bce0247-4a53-4c8b-9c84-152d2b268905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.032908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.032726 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-kube-api-access-g4wgq" pod "7bce0247-4a53-4c8b-9c84-152d2b268905" (UID: "7bce0247-4a53-4c8b-9c84-152d2b268905"). InnerVolumeSpecName "kube-api-access-g4wgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:52.032908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.032848 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-envoy" pod "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" (UID: "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.052067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.051985 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-podinfo" pod "7bce0247-4a53-4c8b-9c84-152d2b268905" (UID: "7bce0247-4a53-4c8b-9c84-152d2b268905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:50:52.056093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.055354 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-envoy" pod "7bce0247-4a53-4c8b-9c84-152d2b268905" (UID: "7bce0247-4a53-4c8b-9c84-152d2b268905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:50:52.056093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.055425 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-token" pod "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" (UID: "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:52.056786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.056733 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-podinfo" pod "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" (UID: "0fba9ea6-4d1f-46e6-bf0f-6dcafe283642"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068772 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068810 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068826 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068842 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068856 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068870 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068885 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068899 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068911 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068926 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068939 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068954 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068968 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bce0247-4a53-4c8b-9c84-152d2b268905-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068983 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdrd\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-kube-api-access-sxdrd\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.068997 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.069011 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bce0247-4a53-4c8b-9c84-152d2b268905-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.069025 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4wgq\" (UniqueName: \"kubernetes.io/projected/7bce0247-4a53-4c8b-9c84-152d2b268905-kube-api-access-g4wgq\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.069240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.069126 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bce0247-4a53-4c8b-9c84-152d2b268905-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:52.478676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.478636 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:52.849349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.845361 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp" Nov 03 23:50:52.849515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.849480 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl" Nov 03 23:50:52.876235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.876210 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp"] Nov 03 23:50:52.885248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.885225 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-r8kkp"] Nov 03 23:50:52.904262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.904233 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl"] Nov 03 23:50:52.907678 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:52.907649 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-vzmtl"] Nov 03 23:50:53.297442 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:53.297410 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" path="/var/lib/kubelet/pods/0fba9ea6-4d1f-46e6-bf0f-6dcafe283642/volumes" Nov 03 23:50:53.297951 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:53.297936 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" path="/var/lib/kubelet/pods/7bce0247-4a53-4c8b-9c84-152d2b268905/volumes" Nov 03 23:50:53.544078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:53.544009 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:54.334251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:54.334221 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:55.203524 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:55.202992 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:55.605056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:55.604118 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" podUID="3f1f9716-dbbb-4ffb-a4c7-95844b82909c" containerName="payload-pre-processing" containerID="cri-o://dedb5441fb451fb921ffcf416d2117814fc30a7c8ab101aba8eb1dad35c9cf97" gracePeriod=25 Nov 03 23:50:55.681463 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:55.681429 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:55.818838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:55.818808 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:50:55.944778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:55.941592 6208 generic.go:356] "Generic (PLEG): container finished" podID="3f1f9716-dbbb-4ffb-a4c7-95844b82909c" containerID="dedb5441fb451fb921ffcf416d2117814fc30a7c8ab101aba8eb1dad35c9cf97" exitCode=0 Nov 03 23:50:55.944778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:55.941635 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" event={"ID":"3f1f9716-dbbb-4ffb-a4c7-95844b82909c","Type":"ContainerDied","Data":"dedb5441fb451fb921ffcf416d2117814fc30a7c8ab101aba8eb1dad35c9cf97"} Nov 03 23:50:56.025055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.024236 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:56.116584 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.116561 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:56.127007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.126977 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-kube-api-access-5vrtn\" (UniqueName: \"kubernetes.io/projected/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-kube-api-access-5vrtn\") pod \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\" (UID: \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\") " Nov 03 23:50:56.129594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.129565 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-kube-api-access-5vrtn" pod "3f1f9716-dbbb-4ffb-a4c7-95844b82909c" (UID: "3f1f9716-dbbb-4ffb-a4c7-95844b82909c"). InnerVolumeSpecName "kube-api-access-5vrtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:50:56.205276 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.201624 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:56.235143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.234020 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-plugins-config-volume\") pod \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\" (UID: \"3f1f9716-dbbb-4ffb-a4c7-95844b82909c\") " Nov 03 23:50:56.235143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.234854 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vrtn\" (UniqueName: \"kubernetes.io/projected/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-kube-api-access-5vrtn\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:56.235626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.235580 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-plugins-config-volume" pod "3f1f9716-dbbb-4ffb-a4c7-95844b82909c" (UID: "3f1f9716-dbbb-4ffb-a4c7-95844b82909c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:50:56.335880 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.335849 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1f9716-dbbb-4ffb-a4c7-95844b82909c-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:50:56.921454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.921416 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:56.950824 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.950799 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" Nov 03 23:50:56.950976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.950832 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6" event={"ID":"3f1f9716-dbbb-4ffb-a4c7-95844b82909c","Type":"ContainerDied","Data":"3386d6ddf1923b9471900bf62d3c8c16cce4eda4389c4c9238b48e048c41e150"} Nov 03 23:50:56.950976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:56.950909 6208 scope.go:122] "RemoveContainer" containerID="dedb5441fb451fb921ffcf416d2117814fc30a7c8ab101aba8eb1dad35c9cf97" Nov 03 23:50:57.032059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.028577 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:57.129493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.129452 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6"] Nov 03 23:50:57.135451 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.135414 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c20f75-6b76b49d79-cjhj6"] Nov 03 23:50:57.312390 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.312356 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1f9716-dbbb-4ffb-a4c7-95844b82909c" path="/var/lib/kubelet/pods/3f1f9716-dbbb-4ffb-a4c7-95844b82909c/volumes" Nov 03 23:50:57.445464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.445427 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:57.668124 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.667890 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc"] Nov 03 23:50:57.735059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.734558 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3f1f9716-dbbb-4ffb-a4c7-95844b82909c" containerName="payload-pre-processing" Nov 03 23:50:57.735059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.734589 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1f9716-dbbb-4ffb-a4c7-95844b82909c" containerName="payload-pre-processing" Nov 03 23:50:57.735059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.734645 6208 cpu_manager.go:414] "containerMap: removing container" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" Nov 03 23:50:57.735059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.734654 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" Nov 03 23:50:57.735059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.734671 6208 cpu_manager.go:414] "containerMap: removing container" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" Nov 03 23:50:57.735059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.734679 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" Nov 03 23:50:57.735059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.734836 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1f9716-dbbb-4ffb-a4c7-95844b82909c" containerName="payload-pre-processing" Nov 03 23:50:57.735059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.734857 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fba9ea6-4d1f-46e6-bf0f-6dcafe283642" containerName="istio-proxy" Nov 03 23:50:57.735059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.734869 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bce0247-4a53-4c8b-9c84-152d2b268905" containerName="istio-proxy" Nov 03 23:50:57.852159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.849812 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:57.931129 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.884871 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc"] Nov 03 23:50:57.939931 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.900057 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:57.952108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.951481 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:57.952108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.951543 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:57.952108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.951677 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:57.952108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.951706 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:57.952108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.951802 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d80a176-9ae6-4d75-8088-26091310280b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:57.952108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.951945 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wbp\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-kube-api-access-j9wbp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:57.952108 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:57.951970 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.065061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.053294 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.065061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.053370 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.065061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.053409 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.065061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.053482 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.065061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.053512 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.065061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.053631 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d80a176-9ae6-4d75-8088-26091310280b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.065061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.053826 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wbp\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-kube-api-access-j9wbp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.065061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.054944 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.110929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.110287 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.110929 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.110623 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.112203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.112157 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d80a176-9ae6-4d75-8088-26091310280b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.114917 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.114777 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.114917 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.114865 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.115068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.114983 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wbp\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-kube-api-access-j9wbp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dhdlc\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.149788 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.149759 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:58.239535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.239480 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:58.651417 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.651386 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc"] Nov 03 23:50:58.660098 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:50:58.653067 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d80a176_9ae6_4d75_8088_26091310280b.slice/crio-d82363982fea48b1dac5888cdc2036439726c746c7dd85354913c32b852f1231 WatchSource:0}: Error finding container d82363982fea48b1dac5888cdc2036439726c746c7dd85354913c32b852f1231: Status 404 returned error can't find the container with id d82363982fea48b1dac5888cdc2036439726c746c7dd85354913c32b852f1231 Nov 03 23:50:58.717059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.714776 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:58.717059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.714872 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:50:58.729913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.729849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:58.841326 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:58.841298 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:59.096891 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:59.096850 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" event={"ID":"0d80a176-9ae6-4d75-8088-26091310280b","Type":"ContainerStarted","Data":"94db4dcc6ee049bb3e0942bb3cab069e52dae3d095925aa44e845fbc8ffa23c0"} Nov 03 23:50:59.100532 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:59.100499 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" event={"ID":"0d80a176-9ae6-4d75-8088-26091310280b","Type":"ContainerStarted","Data":"d82363982fea48b1dac5888cdc2036439726c746c7dd85354913c32b852f1231"} Nov 03 23:50:59.100726 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:59.100709 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:50:59.113409 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:59.113377 6208 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dhdlc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:50:59.113658 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:59.113635 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" podUID="0d80a176-9ae6-4d75-8088-26091310280b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:50:59.165059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:59.157320 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" podStartSLOduration=2.157301429 podStartE2EDuration="2.157301429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:50:59.156004929 +0000 UTC m=+1277.978601094" watchObservedRunningTime="2026-11-03 23:50:59.157301429 +0000 UTC m=+1277.979897600" Nov 03 23:50:59.557688 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:59.557547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:50:59.738366 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:59.738297 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-4dsdb_c07df26b-e714-4555-a633-fa616bb9be2e/payload-processing/0.log" Nov 03 23:50:59.957820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:50:59.957786 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-4dsdb_c07df26b-e714-4555-a633-fa616bb9be2e/payload-processing/0.log" Nov 03 23:51:00.109344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:00.109256 6208 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dhdlc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:51:00.109344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:00.109323 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" podUID="0d80a176-9ae6-4d75-8088-26091310280b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:51:00.172298 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:00.172175 6208 patch_prober.go:29] interesting pod/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:51:00.172298 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:00.172236 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:51:00.715899 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:00.714548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:01.302028 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:01.301993 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:01.684086 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:01.683981 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:01.792346 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:01.792310 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:01.890305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:01.890246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:02.938624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:02.938572 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:03.113816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:03.113779 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 03 23:51:03.234358 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:03.234255 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:03.340515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:03.340442 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:04.101099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:04.101059 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:05.030893 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.030856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:05.163005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.162925 6208 generic.go:356] "Generic (PLEG): container finished" podID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerID="1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7" exitCode=0 Nov 03 23:51:05.163395 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.163000 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" event={"ID":"3131912f-efdb-4051-8ef0-380a18b10bc1","Type":"ContainerDied","Data":"1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7"} Nov 03 23:51:05.163526 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.163507 6208 scope.go:122] "RemoveContainer" containerID="1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7" Nov 03 23:51:05.164857 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.164832 6208 generic.go:356] "Generic (PLEG): container finished" podID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerID="8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf" exitCode=0 Nov 03 23:51:05.164942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.164902 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" event={"ID":"5ad6532c-46b9-439c-88d3-395fe0b78a50","Type":"ContainerDied","Data":"8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf"} Nov 03 23:51:05.165266 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.165251 6208 scope.go:122] "RemoveContainer" containerID="8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf" Nov 03 23:51:05.165343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.165319 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:51:05.165398 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.165385 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:51:05.165439 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.165419 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:51:05.166601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.166576 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:51:05.166667 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.166641 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:51:05.166706 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.166677 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:51:05.706359 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:51:05.706323 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:51:05.838772 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.838740 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:05.943594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:05.943474 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:06.052255 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.052095 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:06.206602 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.206547 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" event={"ID":"3131912f-efdb-4051-8ef0-380a18b10bc1","Type":"ContainerStarted","Data":"d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8"} Nov 03 23:51:06.209112 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.209085 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" event={"ID":"5ad6532c-46b9-439c-88d3-395fe0b78a50","Type":"ContainerStarted","Data":"ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660"} Nov 03 23:51:06.384247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.384152 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:06.419406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.419363 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:51:06.419606 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.419416 6208 patch_prober.go:29] interesting pod/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 23:51:06.419606 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.419467 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 23:51:06.641369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.641271 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:51:06.642508 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.642483 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:51:06.642620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.642547 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:51:06.743193 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:06.743163 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:07.042837 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:07.042803 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:07.420105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:07.419997 6208 patch_prober.go:29] interesting pod/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 23:51:07.420105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:07.420086 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 23:51:07.489057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:07.477097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:07.641298 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:07.641246 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:51:07.641506 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:07.641322 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:51:08.419349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:08.419312 6208 patch_prober.go:29] interesting pod/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 23:51:08.419539 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:08.419371 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 23:51:08.432568 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:08.432525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:08.640763 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:08.640728 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:51:08.640940 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:08.640782 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:51:09.291821 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:09.291778 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:09.430062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:09.426159 6208 patch_prober.go:29] interesting pod/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 23:51:09.430062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:09.426230 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 23:51:09.641754 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:09.641570 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:51:09.641754 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:09.641630 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:51:10.260163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:10.260126 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:51:10.421082 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:10.421021 6208 patch_prober.go:29] interesting pod/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 23:51:10.421283 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:10.421154 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 23:51:10.439301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:10.439155 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:10.642545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:10.642440 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:51:10.642545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:10.642504 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:51:11.420324 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:11.420279 6208 patch_prober.go:29] interesting pod/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 23:51:11.420534 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:11.420337 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 23:51:11.446446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:11.446417 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:11.641629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:11.641587 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:51:11.641804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:11.641661 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:51:12.201822 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.201782 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:12.301155 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.301118 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:12.405851 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.405813 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:12.424504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.424467 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:51:12.425050 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.425012 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:51:12.425765 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.425745 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:51:12.583274 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.583196 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:12.646665 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.646632 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:51:12.647817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.647797 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:51:12.962463 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:12.962413 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:13.096262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:13.096227 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:13.754249 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:13.754217 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:14.759058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:14.758656 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:14.863318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:14.863280 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:15.155158 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:15.155080 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:15.541815 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:15.541775 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:15.667916 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:15.667878 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:16.403153 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:16.403116 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:16.903742 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:16.903707 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:17.001482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:17.001441 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:18.028766 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:18.028733 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:18.151835 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:18.151798 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:18.254318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:18.254284 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:18.957916 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:18.957883 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:19.566205 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:19.566174 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:19.977266 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:19.977237 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:20.830962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:20.830923 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:21.532045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:21.532009 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:22.257382 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:22.257343 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:22.611915 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:22.611830 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:22.710985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:22.710949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:23.102684 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:23.102647 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:23.205828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:23.205791 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:23.545905 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:23.545874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:24.347562 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:24.347529 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:24.447479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:24.447447 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:25.143092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:25.143060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:25.940856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:25.940819 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:26.159773 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:26.159735 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:26.900505 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:26.900449 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:27.739987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:27.739954 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:28.554925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:28.554875 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:28.660874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:28.655115 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:28.759092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:28.759018 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:29.767864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:29.767478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:29.873333 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:29.873298 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:30.250159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:30.250114 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7"] Nov 03 23:51:30.250401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:30.250365 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" containerID="cri-o://c5e05dcec57c9a69e12754119dc375047bf417d627007ed39882590613af5996" gracePeriod=30 Nov 03 23:51:32.438125 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:32.438089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:32.541478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:32.541437 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:32.820210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:32.820168 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:33.620235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:33.620178 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:34.347344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:34.347308 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:35.251780 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.251738 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:35.527210 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.527173 6208 generic.go:356] "Generic (PLEG): container finished" podID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerID="c5e05dcec57c9a69e12754119dc375047bf417d627007ed39882590613af5996" exitCode=0 Nov 03 23:51:35.527384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.527246 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" event={"ID":"33ac3905-17a7-4231-b79a-6cea8e0656b8","Type":"ContainerDied","Data":"c5e05dcec57c9a69e12754119dc375047bf417d627007ed39882590613af5996"} Nov 03 23:51:35.527384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.527318 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" event={"ID":"33ac3905-17a7-4231-b79a-6cea8e0656b8","Type":"ContainerDied","Data":"cac591623c7cc10240457126ce5a13a2e887c89a2fe979b0776f268beb14ab0e"} Nov 03 23:51:35.527384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.527342 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac591623c7cc10240457126ce5a13a2e887c89a2fe979b0776f268beb14ab0e" Nov 03 23:51:35.538005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.530986 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:51:35.658056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.654544 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-podinfo\") pod \"33ac3905-17a7-4231-b79a-6cea8e0656b8\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " Nov 03 23:51:35.658056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.654609 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-data\") pod \"33ac3905-17a7-4231-b79a-6cea8e0656b8\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " Nov 03 23:51:35.658056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.654646 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-socket\") pod \"33ac3905-17a7-4231-b79a-6cea8e0656b8\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " Nov 03 23:51:35.658056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.654681 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-credential-socket\") pod \"33ac3905-17a7-4231-b79a-6cea8e0656b8\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " Nov 03 23:51:35.658056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.654855 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ac3905-17a7-4231-b79a-6cea8e0656b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ac3905-17a7-4231-b79a-6cea8e0656b8-istiod-ca-cert\") pod \"33ac3905-17a7-4231-b79a-6cea8e0656b8\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " Nov 03 23:51:35.658056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.654890 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-envoy\") pod \"33ac3905-17a7-4231-b79a-6cea8e0656b8\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " Nov 03 23:51:35.658056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.655048 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-certs\") pod \"33ac3905-17a7-4231-b79a-6cea8e0656b8\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " Nov 03 23:51:35.658056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.655149 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-token\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-token\") pod \"33ac3905-17a7-4231-b79a-6cea8e0656b8\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " Nov 03 23:51:35.658056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.655239 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-kube-api-access-qpr8f\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-kube-api-access-qpr8f\") pod \"33ac3905-17a7-4231-b79a-6cea8e0656b8\" (UID: \"33ac3905-17a7-4231-b79a-6cea8e0656b8\") " Nov 03 23:51:35.658709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.658529 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-data" pod "33ac3905-17a7-4231-b79a-6cea8e0656b8" (UID: "33ac3905-17a7-4231-b79a-6cea8e0656b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:51:35.659005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.658973 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ac3905-17a7-4231-b79a-6cea8e0656b8-istiod-ca-cert" pod "33ac3905-17a7-4231-b79a-6cea8e0656b8" (UID: "33ac3905-17a7-4231-b79a-6cea8e0656b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:51:35.659277 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.659253 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-socket" pod "33ac3905-17a7-4231-b79a-6cea8e0656b8" (UID: "33ac3905-17a7-4231-b79a-6cea8e0656b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:51:35.659645 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.659625 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-certs" pod "33ac3905-17a7-4231-b79a-6cea8e0656b8" (UID: "33ac3905-17a7-4231-b79a-6cea8e0656b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:51:35.660322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.660295 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-credential-socket" pod "33ac3905-17a7-4231-b79a-6cea8e0656b8" (UID: "33ac3905-17a7-4231-b79a-6cea8e0656b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:51:35.661330 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.661306 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ac3905-17a7-4231-b79a-6cea8e0656b8-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:35.661467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.661337 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:35.661467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.661352 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:35.661467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.661365 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:35.661467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.661377 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:35.667548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.667394 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-podinfo" pod "33ac3905-17a7-4231-b79a-6cea8e0656b8" (UID: "33ac3905-17a7-4231-b79a-6cea8e0656b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:51:35.667548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.667517 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-envoy" pod "33ac3905-17a7-4231-b79a-6cea8e0656b8" (UID: "33ac3905-17a7-4231-b79a-6cea8e0656b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:51:35.679078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.679028 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-token" pod "33ac3905-17a7-4231-b79a-6cea8e0656b8" (UID: "33ac3905-17a7-4231-b79a-6cea8e0656b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:51:35.687644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.687613 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-kube-api-access-qpr8f" pod "33ac3905-17a7-4231-b79a-6cea8e0656b8" (UID: "33ac3905-17a7-4231-b79a-6cea8e0656b8"). InnerVolumeSpecName "kube-api-access-qpr8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:51:35.714485 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:51:35.714440 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:51:35.765214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.764554 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:35.765214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.764637 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpr8f\" (UniqueName: \"kubernetes.io/projected/33ac3905-17a7-4231-b79a-6cea8e0656b8-kube-api-access-qpr8f\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:35.765214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.764656 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:35.765214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:35.764692 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ac3905-17a7-4231-b79a-6cea8e0656b8-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:36.050021 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:36.049917 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:36.535687 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:36.535654 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7" Nov 03 23:51:36.564019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:36.563982 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7"] Nov 03 23:51:36.568226 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:36.568196 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-kb7j7"] Nov 03 23:51:36.949656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:36.949567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:37.295816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:37.295789 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" path="/var/lib/kubelet/pods/33ac3905-17a7-4231-b79a-6cea8e0656b8/volumes" Nov 03 23:51:37.745265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:37.745229 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:38.491876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:38.491837 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:38.974849 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:38.974817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:39.422913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:39.422798 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:40.353860 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:40.353812 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:41.150751 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:41.150717 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:41.918966 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:41.918927 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:42.016642 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:42.016607 6208 scope.go:122] "RemoveContainer" containerID="c974345fc6e21bf1dbbb62bc345fc24522bd0305f7478255cafd2832facb89cc" Nov 03 23:51:42.029528 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:42.029501 6208 scope.go:122] "RemoveContainer" containerID="e9fff241b73a561520299b96393cbb430295656c35f272800d4b54d897066359" Nov 03 23:51:42.043973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:42.043947 6208 scope.go:122] "RemoveContainer" containerID="9688b0138abe642150a9c15c017e6400e40f18627f37dad785c4f97079e4ace7" Nov 03 23:51:42.054669 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:42.054651 6208 scope.go:122] "RemoveContainer" containerID="4dc03062f3246e8527da0516b18419b6bc9b51c08186d28b48fea6b0f25e4d50" Nov 03 23:51:42.078201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:42.078170 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:43.195894 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:43.195862 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:43.295542 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:43.295490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:44.163883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:44.163845 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:44.841491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:44.841450 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:45.751395 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:45.751354 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:46.076896 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:46.076811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:46.635342 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:46.635309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:47.250045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:47.249996 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:47.394488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:47.394453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:47.526120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:47.525153 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:47.633112 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:47.633017 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:48.185250 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.185217 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s"] Nov 03 23:51:48.185551 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.185523 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" podUID="0c7889d7-d920-4d3a-bd33-8ea7e07041a0" containerName="maas-api" containerID="cri-o://50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c" gracePeriod=30 Nov 03 23:51:48.232876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.232845 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:48.499947 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.499915 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:51:48.574500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.574457 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-maas-api-tls\") pod \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " Nov 03 23:51:48.574704 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.574605 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-openshift-service-ca\") pod \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " Nov 03 23:51:48.574704 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.574656 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-metrics-certs\") pod \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " Nov 03 23:51:48.574825 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.574804 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-kube-api-access-sv845\" (UniqueName: \"kubernetes.io/projected/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-kube-api-access-sv845\") pod \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\" (UID: \"0c7889d7-d920-4d3a-bd33-8ea7e07041a0\") " Nov 03 23:51:48.575352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.575306 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-openshift-service-ca" pod "0c7889d7-d920-4d3a-bd33-8ea7e07041a0" (UID: "0c7889d7-d920-4d3a-bd33-8ea7e07041a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:51:48.575668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.575599 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:48.578431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.578407 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-kube-api-access-sv845" pod "0c7889d7-d920-4d3a-bd33-8ea7e07041a0" (UID: "0c7889d7-d920-4d3a-bd33-8ea7e07041a0"). InnerVolumeSpecName "kube-api-access-sv845". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:51:48.578923 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.578882 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-metrics-certs" pod "0c7889d7-d920-4d3a-bd33-8ea7e07041a0" (UID: "0c7889d7-d920-4d3a-bd33-8ea7e07041a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:51:48.584194 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.584167 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-maas-api-tls" pod "0c7889d7-d920-4d3a-bd33-8ea7e07041a0" (UID: "0c7889d7-d920-4d3a-bd33-8ea7e07041a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:51:48.639767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.639735 6208 generic.go:356] "Generic (PLEG): container finished" podID="0c7889d7-d920-4d3a-bd33-8ea7e07041a0" containerID="50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c" exitCode=0 Nov 03 23:51:48.639945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.639782 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" Nov 03 23:51:48.639945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.639824 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" event={"ID":"0c7889d7-d920-4d3a-bd33-8ea7e07041a0","Type":"ContainerDied","Data":"50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c"} Nov 03 23:51:48.639945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.639883 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s" event={"ID":"0c7889d7-d920-4d3a-bd33-8ea7e07041a0","Type":"ContainerDied","Data":"42236781cf460c3cf9b7c9cce6d7dd06c2d016b28b07e0ec0303667d95b8e4bb"} Nov 03 23:51:48.639945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.639913 6208 scope.go:122] "RemoveContainer" containerID="50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c" Nov 03 23:51:48.652724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.652663 6208 scope.go:122] "RemoveContainer" containerID="50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c" Nov 03 23:51:48.653047 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:51:48.653006 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c\": container with ID starting with 50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c not found: ID does not exist" containerID="50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c" Nov 03 23:51:48.653156 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.653052 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c"} err="failed to get container status \"50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c\": rpc error: code = NotFound desc = could not find container \"50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c\": container with ID starting with 50495249be2b7a011dd168969175ee1f6adb6b653f15c22054dc2a5e0fa9ce7c not found: ID does not exist" Nov 03 23:51:48.669408 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.669373 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s"] Nov 03 23:51:48.673180 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.673150 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c83d52-5f4684d5b4-drw2s"] Nov 03 23:51:48.677275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.677252 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:48.677275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.677276 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:48.677425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.677286 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv845\" (UniqueName: \"kubernetes.io/projected/0c7889d7-d920-4d3a-bd33-8ea7e07041a0-kube-api-access-sv845\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:51:48.694728 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:48.694690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:49.297198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:49.297166 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7889d7-d920-4d3a-bd33-8ea7e07041a0" path="/var/lib/kubelet/pods/0c7889d7-d920-4d3a-bd33-8ea7e07041a0/volumes" Nov 03 23:51:49.547842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:49.547753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:50.613504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:50.613464 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:51.442778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:51.442738 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:52.329189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:52.329159 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:52.429012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:52.428972 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:53.066576 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:53.066543 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:53.597945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:53.597909 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:53.696362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:53.696322 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:54.256359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:54.256328 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:55.163741 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:55.163706 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:55.894269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:55.894224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:55.994014 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:55.993984 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:56.100565 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:56.100533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:56.547786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:56.547745 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:56.798646 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:56.798565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:56.899886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:56.899847 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:57.246710 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:57.246678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:58.236754 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:58.236716 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:59.030423 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:59.030381 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:59.738188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:59.738151 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:51:59.980246 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:51:59.980205 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:00.838608 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:00.838573 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:01.684924 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:01.684832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:02.437312 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:02.437271 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:03.133655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:03.133612 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:03.228528 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:03.228495 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:03.354152 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:03.354105 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:04.268514 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.268473 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb"] Nov 03 23:52:04.269302 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.269270 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" podUID="a3cb46f5-7aa2-463f-b606-9e9744603ce5" containerName="maas-api" containerID="cri-o://f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971" gracePeriod=30 Nov 03 23:52:04.315327 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.315287 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:04.433318 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.433276 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:04.534494 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.534455 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:04.558644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.558622 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:52:04.758493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.758457 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-maas-api-tls\") pod \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " Nov 03 23:52:04.759006 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.758985 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3cb46f5-7aa2-463f-b606-9e9744603ce5-kube-api-access-9d2gj\" (UniqueName: \"kubernetes.io/projected/a3cb46f5-7aa2-463f-b606-9e9744603ce5-kube-api-access-9d2gj\") pod \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " Nov 03 23:52:04.759479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.759459 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3cb46f5-7aa2-463f-b606-9e9744603ce5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cb46f5-7aa2-463f-b606-9e9744603ce5-openshift-service-ca\") pod \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " Nov 03 23:52:04.759987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.759972 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-metrics-certs\") pod \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\" (UID: \"a3cb46f5-7aa2-463f-b606-9e9744603ce5\") " Nov 03 23:52:04.760359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.759904 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3cb46f5-7aa2-463f-b606-9e9744603ce5-openshift-service-ca" pod "a3cb46f5-7aa2-463f-b606-9e9744603ce5" (UID: "a3cb46f5-7aa2-463f-b606-9e9744603ce5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:52:04.762582 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.762562 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cb46f5-7aa2-463f-b606-9e9744603ce5-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:04.763008 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.762977 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-maas-api-tls" pod "a3cb46f5-7aa2-463f-b606-9e9744603ce5" (UID: "a3cb46f5-7aa2-463f-b606-9e9744603ce5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:52:04.766878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.766195 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cb46f5-7aa2-463f-b606-9e9744603ce5-kube-api-access-9d2gj" pod "a3cb46f5-7aa2-463f-b606-9e9744603ce5" (UID: "a3cb46f5-7aa2-463f-b606-9e9744603ce5"). InnerVolumeSpecName "kube-api-access-9d2gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:52:04.767983 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.767955 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-metrics-certs" pod "a3cb46f5-7aa2-463f-b606-9e9744603ce5" (UID: "a3cb46f5-7aa2-463f-b606-9e9744603ce5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:52:04.785023 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.784955 6208 generic.go:356] "Generic (PLEG): container finished" podID="a3cb46f5-7aa2-463f-b606-9e9744603ce5" containerID="f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971" exitCode=0 Nov 03 23:52:04.786449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.785008 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" Nov 03 23:52:04.786449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.785026 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" event={"ID":"a3cb46f5-7aa2-463f-b606-9e9744603ce5","Type":"ContainerDied","Data":"f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971"} Nov 03 23:52:04.786449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.786106 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb" event={"ID":"a3cb46f5-7aa2-463f-b606-9e9744603ce5","Type":"ContainerDied","Data":"2d275c42513220d59ac98e4daf59cadb826054dd2fa32cd399657644d8a6b18e"} Nov 03 23:52:04.786449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.786145 6208 scope.go:122] "RemoveContainer" containerID="f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971" Nov 03 23:52:04.795280 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.795257 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:04.802282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.802261 6208 scope.go:122] "RemoveContainer" containerID="f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971" Nov 03 23:52:04.802599 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:52:04.802564 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971\": container with ID starting with f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971 not found: ID does not exist" containerID="f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971" Nov 03 23:52:04.802689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.802609 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971"} err="failed to get container status \"f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971\": rpc error: code = NotFound desc = could not find container \"f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971\": container with ID starting with f0338d7372b03262bb0b858eab35bd0e59a2ebd43a97b5b64e07c7b75a6fc971 not found: ID does not exist" Nov 03 23:52:04.824617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.822130 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb"] Nov 03 23:52:04.825885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.825857 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4892d-8698b99889-cctmb"] Nov 03 23:52:04.864515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.864485 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2gj\" (UniqueName: \"kubernetes.io/projected/a3cb46f5-7aa2-463f-b606-9e9744603ce5-kube-api-access-9d2gj\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:04.864515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.864513 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:04.864676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.864523 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cb46f5-7aa2-463f-b606-9e9744603ce5-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:04.949527 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:04.949493 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:05.296051 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:05.296002 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3cb46f5-7aa2-463f-b606-9e9744603ce5" path="/var/lib/kubelet/pods/a3cb46f5-7aa2-463f-b606-9e9744603ce5/volumes" Nov 03 23:52:05.708972 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:52:05.708940 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:52:05.756219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:05.756191 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:06.031106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:06.030980 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:06.748844 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:06.748798 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:06.848907 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:06.848863 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:07.118704 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:07.118619 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:07.324901 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:07.324861 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:07.704919 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:07.704884 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:08.241237 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:08.241200 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:08.955491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:08.955454 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:09.589377 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:09.589342 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:09.702301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:09.702247 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:10.008091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:10.008028 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:10.844822 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:10.844793 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:11.532972 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:11.532932 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:11.778415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:11.778378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:11.878771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:11.878637 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:12.438659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:12.438599 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:12.552264 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:12.552214 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:12.790697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:12.790658 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l"] Nov 03 23:52:12.790955 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:12.790929 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" containerID="cri-o://d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8" gracePeriod=30 Nov 03 23:52:12.952965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:12.952925 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:13.516995 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:13.516960 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:13.667668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:13.667637 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:14.332812 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:14.332777 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:15.093489 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:15.093454 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:15.994349 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:15.994312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:16.470316 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:16.470281 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:16.983881 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:16.983836 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:17.335991 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:17.335906 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:18.041285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.041246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:18.070427 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.070360 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:52:18.093582 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.093502 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-kube-api-access-rm58c\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-kube-api-access-rm58c\") pod \"3131912f-efdb-4051-8ef0-380a18b10bc1\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " Nov 03 23:52:18.093989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.093879 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-envoy\") pod \"3131912f-efdb-4051-8ef0-380a18b10bc1\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " Nov 03 23:52:18.093989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.093958 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-socket\") pod \"3131912f-efdb-4051-8ef0-380a18b10bc1\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " Nov 03 23:52:18.094170 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.094058 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-credential-socket\") pod \"3131912f-efdb-4051-8ef0-380a18b10bc1\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " Nov 03 23:52:18.094170 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.094118 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-certs\") pod \"3131912f-efdb-4051-8ef0-380a18b10bc1\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " Nov 03 23:52:18.094273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.094210 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-data\") pod \"3131912f-efdb-4051-8ef0-380a18b10bc1\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " Nov 03 23:52:18.094273 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.094245 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-podinfo\") pod \"3131912f-efdb-4051-8ef0-380a18b10bc1\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " Nov 03 23:52:18.094508 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.094479 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-token\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-token\") pod \"3131912f-efdb-4051-8ef0-380a18b10bc1\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " Nov 03 23:52:18.094635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.094520 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3131912f-efdb-4051-8ef0-380a18b10bc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3131912f-efdb-4051-8ef0-380a18b10bc1-istiod-ca-cert\") pod \"3131912f-efdb-4051-8ef0-380a18b10bc1\" (UID: \"3131912f-efdb-4051-8ef0-380a18b10bc1\") " Nov 03 23:52:18.094711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.094680 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-credential-socket" pod "3131912f-efdb-4051-8ef0-380a18b10bc1" (UID: "3131912f-efdb-4051-8ef0-380a18b10bc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.095203 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-certs" pod "3131912f-efdb-4051-8ef0-380a18b10bc1" (UID: "3131912f-efdb-4051-8ef0-380a18b10bc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.095403 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-socket" pod "3131912f-efdb-4051-8ef0-380a18b10bc1" (UID: "3131912f-efdb-4051-8ef0-380a18b10bc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.095480 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-data" pod "3131912f-efdb-4051-8ef0-380a18b10bc1" (UID: "3131912f-efdb-4051-8ef0-380a18b10bc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.097094 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3131912f-efdb-4051-8ef0-380a18b10bc1-istiod-ca-cert" pod "3131912f-efdb-4051-8ef0-380a18b10bc1" (UID: "3131912f-efdb-4051-8ef0-380a18b10bc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.097540 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.097561 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.097576 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.097592 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.097605 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3131912f-efdb-4051-8ef0-380a18b10bc1-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.100703 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-kube-api-access-rm58c" pod "3131912f-efdb-4051-8ef0-380a18b10bc1" (UID: "3131912f-efdb-4051-8ef0-380a18b10bc1"). InnerVolumeSpecName "kube-api-access-rm58c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.101058 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-envoy" pod "3131912f-efdb-4051-8ef0-380a18b10bc1" (UID: "3131912f-efdb-4051-8ef0-380a18b10bc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:18.103092 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.102066 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-podinfo" pod "3131912f-efdb-4051-8ef0-380a18b10bc1" (UID: "3131912f-efdb-4051-8ef0-380a18b10bc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:52:18.110895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.109435 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-token" pod "3131912f-efdb-4051-8ef0-380a18b10bc1" (UID: "3131912f-efdb-4051-8ef0-380a18b10bc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:52:18.199012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.198968 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:18.199012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.198999 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:18.199012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.199010 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm58c\" (UniqueName: \"kubernetes.io/projected/3131912f-efdb-4051-8ef0-380a18b10bc1-kube-api-access-rm58c\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:18.199012 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.199020 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3131912f-efdb-4051-8ef0-380a18b10bc1-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:18.906653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.906612 6208 generic.go:356] "Generic (PLEG): container finished" podID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerID="d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8" exitCode=0 Nov 03 23:52:18.907159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.906667 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" Nov 03 23:52:18.907159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.906676 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" event={"ID":"3131912f-efdb-4051-8ef0-380a18b10bc1","Type":"ContainerDied","Data":"d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8"} Nov 03 23:52:18.907159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.906727 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l" event={"ID":"3131912f-efdb-4051-8ef0-380a18b10bc1","Type":"ContainerDied","Data":"883ad02bc01d4167c767acf75fff637b3e4f017747d0a8ee6830306aa6df5eff"} Nov 03 23:52:18.907159 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.906757 6208 scope.go:122] "RemoveContainer" containerID="d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8" Nov 03 23:52:18.918106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.918080 6208 scope.go:122] "RemoveContainer" containerID="1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7" Nov 03 23:52:18.927802 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.927777 6208 scope.go:122] "RemoveContainer" containerID="d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8" Nov 03 23:52:18.928154 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:52:18.928020 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8\": container with ID starting with d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8 not found: ID does not exist" containerID="d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8" Nov 03 23:52:18.928230 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.928164 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8"} err="failed to get container status \"d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8\": rpc error: code = NotFound desc = could not find container \"d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8\": container with ID starting with d9875c416aa04cf9c7dd7256929150631b494480da322d680b74f3b4d9f8c8e8 not found: ID does not exist" Nov 03 23:52:18.928230 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.928182 6208 scope.go:122] "RemoveContainer" containerID="1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7" Nov 03 23:52:18.928445 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:52:18.928426 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7\": container with ID starting with 1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7 not found: ID does not exist" containerID="1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7" Nov 03 23:52:18.928485 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.928452 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7"} err="failed to get container status \"1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7\": rpc error: code = NotFound desc = could not find container \"1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7\": container with ID starting with 1ad7d7b459f5c1304fb8cbb67e4fbf4ec9e33f34c0798cfae1c56f7d2f0041a7 not found: ID does not exist" Nov 03 23:52:18.935409 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.935382 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l"] Nov 03 23:52:18.938670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.938647 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-695f03-openshift-default-74c67f95bf-z7c9l"] Nov 03 23:52:18.940011 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:18.939985 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:19.296418 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:19.296382 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" path="/var/lib/kubelet/pods/3131912f-efdb-4051-8ef0-380a18b10bc1/volumes" Nov 03 23:52:19.724950 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:19.724922 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:20.409598 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:20.409565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:20.504399 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:20.504360 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:21.250295 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:21.250260 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:21.415865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:21.415832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:22.055881 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:22.055844 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:22.892043 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:22.892005 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:23.497051 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:23.496995 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:23.672584 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:23.672549 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:24.256842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:24.256805 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:24.354879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:24.354844 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:24.540386 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:24.540305 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:25.168873 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:25.168840 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:25.288455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:25.288409 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:25.414943 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:25.414903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:25.513177 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:25.513140 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:26.027345 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:26.027312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:27.235737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:27.235702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:28.317937 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:28.317901 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:28.420370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:28.420334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:29.136105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:29.136064 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:29.836839 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:29.836806 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:30.819983 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:30.819951 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:31.399267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:31.399230 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:31.544291 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:31.544250 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:32.415447 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:32.415414 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:33.252446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:33.252408 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:33.945611 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:33.945576 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:34.053357 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:34.053322 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:34.450567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:34.450535 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:34.833491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:34.833453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:35.127588 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:35.127504 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:35.657722 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:35.657688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:35.705434 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:52:35.705403 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:52:36.008626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:36.008582 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:36.108238 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:36.108206 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:36.208614 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:36.208570 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:36.546326 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:36.546280 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:37.327385 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:37.327352 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:38.118582 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:38.118547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:38.219064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:38.219008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:38.605369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:38.605333 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:38.793117 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:38.793083 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:38.933585 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:38.933505 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l"] Nov 03 23:52:38.933798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:38.933771 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" containerID="cri-o://320e8f63f3736d1476f5507ca37d7441b0542fb883b2a5a199caa27fc9b69bda" gracePeriod=30 Nov 03 23:52:38.962367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:38.962335 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:39.258833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:39.258802 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:39.422600 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:39.422565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:40.073687 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:40.073649 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:40.174760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:40.174728 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:41.046151 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:41.046111 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:41.921332 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:41.921292 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:42.103439 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:42.103408 6208 scope.go:122] "RemoveContainer" containerID="c5e05dcec57c9a69e12754119dc375047bf417d627007ed39882590613af5996" Nov 03 23:52:42.790991 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:42.790957 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:43.172760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:43.172672 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:43.491799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:43.491758 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:43.591868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:43.591829 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:43.693483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:43.693442 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:44.119448 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.119416 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:44.173842 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.173793 6208 generic.go:356] "Generic (PLEG): container finished" podID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerID="320e8f63f3736d1476f5507ca37d7441b0542fb883b2a5a199caa27fc9b69bda" exitCode=0 Nov 03 23:52:44.174375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.174192 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" event={"ID":"12b14f1d-b748-4b8b-a43c-79a6589b84f8","Type":"ContainerDied","Data":"320e8f63f3736d1476f5507ca37d7441b0542fb883b2a5a199caa27fc9b69bda"} Nov 03 23:52:44.174375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.174295 6208 scope.go:122] "RemoveContainer" containerID="ea7d7e28d668e5f9f456c4785e3c1030535a10e06b403dbea76936a895409d84" Nov 03 23:52:44.261888 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.261849 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:52:44.352088 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352057 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-credential-socket\") pod \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " Nov 03 23:52:44.352261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352110 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-socket\") pod \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " Nov 03 23:52:44.352261 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352159 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-kube-api-access-phr9g\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-kube-api-access-phr9g\") pod \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " Nov 03 23:52:44.352339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352276 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-data\") pod \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " Nov 03 23:52:44.352374 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352359 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-envoy\") pod \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " Nov 03 23:52:44.352436 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352378 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-credential-socket" pod "12b14f1d-b748-4b8b-a43c-79a6589b84f8" (UID: "12b14f1d-b748-4b8b-a43c-79a6589b84f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:44.352488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352409 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-token\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-token\") pod \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " Nov 03 23:52:44.352488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352474 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-socket" pod "12b14f1d-b748-4b8b-a43c-79a6589b84f8" (UID: "12b14f1d-b748-4b8b-a43c-79a6589b84f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:44.352575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352556 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-certs\") pod \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " Nov 03 23:52:44.352626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352604 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-data" pod "12b14f1d-b748-4b8b-a43c-79a6589b84f8" (UID: "12b14f1d-b748-4b8b-a43c-79a6589b84f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:44.352626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352615 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-podinfo\") pod \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " Nov 03 23:52:44.352783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352764 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istiod-ca-cert\") pod \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\" (UID: \"12b14f1d-b748-4b8b-a43c-79a6589b84f8\") " Nov 03 23:52:44.352875 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.352853 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-certs" pod "12b14f1d-b748-4b8b-a43c-79a6589b84f8" (UID: "12b14f1d-b748-4b8b-a43c-79a6589b84f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:44.353219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.353150 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istiod-ca-cert" pod "12b14f1d-b748-4b8b-a43c-79a6589b84f8" (UID: "12b14f1d-b748-4b8b-a43c-79a6589b84f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:52:44.354163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.353788 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:44.354163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.353814 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:44.354163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.353829 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:44.354163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.353843 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:44.354163 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.353856 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:44.354839 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.354791 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-podinfo" pod "12b14f1d-b748-4b8b-a43c-79a6589b84f8" (UID: "12b14f1d-b748-4b8b-a43c-79a6589b84f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:52:44.354942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.354912 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-kube-api-access-phr9g" pod "12b14f1d-b748-4b8b-a43c-79a6589b84f8" (UID: "12b14f1d-b748-4b8b-a43c-79a6589b84f8"). InnerVolumeSpecName "kube-api-access-phr9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:52:44.355025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.355005 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-token" pod "12b14f1d-b748-4b8b-a43c-79a6589b84f8" (UID: "12b14f1d-b748-4b8b-a43c-79a6589b84f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:52:44.355310 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.355289 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-envoy" pod "12b14f1d-b748-4b8b-a43c-79a6589b84f8" (UID: "12b14f1d-b748-4b8b-a43c-79a6589b84f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:52:44.454921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.454884 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phr9g\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-kube-api-access-phr9g\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:44.454921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.454917 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:44.454921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.454927 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:44.455175 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:44.454935 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12b14f1d-b748-4b8b-a43c-79a6589b84f8-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:52:45.128449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:45.128408 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:45.183253 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:45.183212 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" event={"ID":"12b14f1d-b748-4b8b-a43c-79a6589b84f8","Type":"ContainerDied","Data":"d2e912c6092c4cbd581136f3221489cd8b17c30a7ad34c084da5e6b86fb764f9"} Nov 03 23:52:45.183253 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:45.183242 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l" Nov 03 23:52:45.183732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:45.183281 6208 scope.go:122] "RemoveContainer" containerID="320e8f63f3736d1476f5507ca37d7441b0542fb883b2a5a199caa27fc9b69bda" Nov 03 23:52:45.209895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:45.209867 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l"] Nov 03 23:52:45.217700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:45.217668 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c887c7-openshift-default-9c7f885d5-88j7l"] Nov 03 23:52:45.226237 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:45.226206 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:45.304348 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:45.304312 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" path="/var/lib/kubelet/pods/12b14f1d-b748-4b8b-a43c-79a6589b84f8/volumes" Nov 03 23:52:46.226158 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:46.226124 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:47.139153 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:47.139117 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:47.989234 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:47.989179 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:48.876604 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:48.876565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:49.638774 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:49.638739 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:50.496361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:50.496319 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:51.076973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:51.076940 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:51.185510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:51.185471 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:51.425190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:51.425097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:51.746107 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:51.745584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:52.115385 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:52.115306 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:52.833275 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:52.833240 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:53.656821 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:53.656789 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:53.757903 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:53.757869 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:54.140067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:54.140024 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:54.459384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:54.459348 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:54.961935 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:54.961895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:55.843680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:55.843633 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:57.257693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:57.257635 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:57.868123 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:57.868089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:58.096213 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:58.096175 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:58.297462 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:58.297423 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:58.873093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:58.873054 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:59.713877 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:59.713845 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:59.846137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:59.846102 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:52:59.953990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:52:59.953954 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:00.710470 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:00.710429 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:01.515169 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:01.515130 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:02.353560 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:02.353532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:03.151496 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:03.151458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:03.941532 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:03.941486 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:04.840789 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:04.840734 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:04.941491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:04.941458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:05.706748 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:53:05.706708 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:53:05.743614 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:05.743579 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:06.094664 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:06.094569 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:06.583919 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:06.583880 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:07.341752 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:07.341714 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:08.152708 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:08.152670 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:08.887976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:08.887925 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:08.948717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:08.948677 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z"] Nov 03 23:53:08.948973 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:08.948943 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" containerID="cri-o://289230f4ef2e10a7a06419fa4eb552f73dd70598199c41dc4e8f36475c2ea091" gracePeriod=30 Nov 03 23:53:09.626754 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:09.626715 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:10.337355 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:10.337324 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:11.146266 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:11.146224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:12.022091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:12.022050 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:13.652633 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:13.652577 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:13.877980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:13.877934 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:14.503313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:14.503262 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 23:53:14.540529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:14.540484 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:14.598430 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:14.598380 6208 generic.go:356] "Generic (PLEG): container finished" podID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerID="289230f4ef2e10a7a06419fa4eb552f73dd70598199c41dc4e8f36475c2ea091" exitCode=0 Nov 03 23:53:14.598962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:14.598680 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" event={"ID":"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf","Type":"ContainerDied","Data":"289230f4ef2e10a7a06419fa4eb552f73dd70598199c41dc4e8f36475c2ea091"} Nov 03 23:53:14.989998 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:14.989969 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:53:15.128660 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.128619 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-podinfo\") pod \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " Nov 03 23:53:15.128862 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.128745 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-kube-api-access-g6h9l\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-kube-api-access-g6h9l\") pod \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " Nov 03 23:53:15.128862 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.128786 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istiod-ca-cert\") pod \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " Nov 03 23:53:15.128993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.128864 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-token\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-token\") pod \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " Nov 03 23:53:15.128993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.128898 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-certs\") pod \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " Nov 03 23:53:15.128993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.128949 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-credential-socket\") pod \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " Nov 03 23:53:15.129185 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129076 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-data\") pod \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " Nov 03 23:53:15.129185 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129134 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-socket\") pod \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " Nov 03 23:53:15.129293 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129223 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-certs" pod "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" (UID: "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:53:15.129293 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129240 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-envoy\") pod \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\" (UID: \"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf\") " Nov 03 23:53:15.129402 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129348 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-credential-socket" pod "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" (UID: "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:53:15.129402 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129375 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istiod-ca-cert" pod "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" (UID: "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:53:15.129519 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129454 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-data" pod "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" (UID: "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:53:15.129590 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129569 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-socket" pod "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" (UID: "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:53:15.130245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129849 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:53:15.130245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129879 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:53:15.130245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129894 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:53:15.130245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129910 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:53:15.130245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.129924 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:53:15.136168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.135973 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-kube-api-access-g6h9l" pod "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" (UID: "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf"). InnerVolumeSpecName "kube-api-access-g6h9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:53:15.136698 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.136670 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-podinfo" pod "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" (UID: "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:53:15.138656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.138599 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-envoy" pod "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" (UID: "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:53:15.139150 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.139127 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-token" pod "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" (UID: "d6cd0f90-1f9e-4622-a550-fdde21f0e4cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:53:15.231203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.231167 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:53:15.231203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.231195 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:53:15.231203 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.231204 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:53:15.231495 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.231214 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6h9l\" (UniqueName: \"kubernetes.io/projected/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf-kube-api-access-g6h9l\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:53:15.607511 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.607427 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" Nov 03 23:53:15.607511 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.607469 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z" event={"ID":"d6cd0f90-1f9e-4622-a550-fdde21f0e4cf","Type":"ContainerDied","Data":"5d25f0040b8f71ac501694508fdb27314d1b972b82b66a06a0c20b4fc970a55a"} Nov 03 23:53:15.607724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.607530 6208 scope.go:122] "RemoveContainer" containerID="289230f4ef2e10a7a06419fa4eb552f73dd70598199c41dc4e8f36475c2ea091" Nov 03 23:53:15.633475 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.633446 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z"] Nov 03 23:53:15.641258 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.641234 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-cnl6z"] Nov 03 23:53:15.924906 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:15.924809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:16.023895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:16.023844 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:16.732888 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:16.732851 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:17.297548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:17.297516 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" path="/var/lib/kubelet/pods/d6cd0f90-1f9e-4622-a550-fdde21f0e4cf/volumes" Nov 03 23:53:17.728580 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:17.728542 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:18.594726 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:18.594690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:19.545725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:19.545691 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:20.547198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:20.547159 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:21.343693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:21.343661 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:22.488689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:22.488645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:23.588352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:23.588315 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:23.717563 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:23.717393 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:23.812948 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:23.812907 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:24.217872 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:24.217828 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:24.921910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:24.921873 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:25.022654 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:25.022619 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:25.797094 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:25.796996 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:26.573245 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:26.573148 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:27.453448 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:27.453395 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:28.443641 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:28.443604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:28.547850 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:28.547812 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:29.341504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:29.341459 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:29.723430 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:29.723398 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:30.163847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.163741 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:30.266609 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.266570 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:30.479181 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479150 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk"] Nov 03 23:53:30.479761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479747 6208 cpu_manager.go:414] "containerMap: removing container" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" Nov 03 23:53:30.479806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479763 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" Nov 03 23:53:30.479806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479772 6208 cpu_manager.go:414] "containerMap: removing container" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" Nov 03 23:53:30.479806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479777 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" Nov 03 23:53:30.479806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479794 6208 cpu_manager.go:414] "containerMap: removing container" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" Nov 03 23:53:30.479806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479800 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479810 6208 cpu_manager.go:414] "containerMap: removing container" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479815 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479832 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479836 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479848 6208 cpu_manager.go:414] "containerMap: removing container" podUID="0c7889d7-d920-4d3a-bd33-8ea7e07041a0" containerName="maas-api" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479854 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7889d7-d920-4d3a-bd33-8ea7e07041a0" containerName="maas-api" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479863 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a3cb46f5-7aa2-463f-b606-9e9744603ce5" containerName="maas-api" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479868 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3cb46f5-7aa2-463f-b606-9e9744603ce5" containerName="maas-api" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479876 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479880 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" Nov 03 23:53:30.479962 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479964 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" Nov 03 23:53:30.480443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479974 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7889d7-d920-4d3a-bd33-8ea7e07041a0" containerName="maas-api" Nov 03 23:53:30.480443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479981 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3cb46f5-7aa2-463f-b606-9e9744603ce5" containerName="maas-api" Nov 03 23:53:30.480443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479988 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3131912f-efdb-4051-8ef0-380a18b10bc1" containerName="istio-proxy" Nov 03 23:53:30.480443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.479996 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" Nov 03 23:53:30.480443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.480001 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b14f1d-b748-4b8b-a43c-79a6589b84f8" containerName="istio-proxy" Nov 03 23:53:30.480443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.480008 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ac3905-17a7-4231-b79a-6cea8e0656b8" containerName="istio-proxy" Nov 03 23:53:30.480443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.480013 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6cd0f90-1f9e-4622-a550-fdde21f0e4cf" containerName="istio-proxy" Nov 03 23:53:30.484235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.484215 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.503779 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.503400 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk"] Nov 03 23:53:30.509601 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.509562 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:30.569802 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.569766 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.569993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.569808 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.569993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.569921 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.569993 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.569983 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59tt\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-kube-api-access-q59tt\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.570204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.570113 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e697bb-1d96-4ee6-aa78-78afce02e434-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.570204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.570151 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.570290 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.570278 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.570330 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.570302 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.570400 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.570337 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.671782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.671749 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.671990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.671793 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.671990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.671862 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.671990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.671969 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59tt\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-kube-api-access-q59tt\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.672224 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.672066 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e697bb-1d96-4ee6-aa78-78afce02e434-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.672224 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.672104 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.672333 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.672232 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.672333 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.672270 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.672333 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.672329 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.672488 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.672395 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-credential-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.672805 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.672780 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-socket\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.672952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.672861 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-certs\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.672952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.672940 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-data\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.673322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.673299 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e697bb-1d96-4ee6-aa78-78afce02e434-istiod-ca-cert\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.675183 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.675158 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-envoy\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.676964 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.676946 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-podinfo\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.685357 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.685332 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-token\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.685451 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.685359 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59tt\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-kube-api-access-q59tt\") pod \"e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:30.812083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:30.811975 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:31.102400 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.102371 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk"] Nov 03 23:53:31.103568 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:53:31.103540 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e697bb_1d96_4ee6_aa78_78afce02e434.slice/crio-9a950dd712ca8f97a509162855419bf4e00d2a32c452232b324fe910eaadab25 WatchSource:0}: Error finding container 9a950dd712ca8f97a509162855419bf4e00d2a32c452232b324fe910eaadab25: Status 404 returned error can't find the container with id 9a950dd712ca8f97a509162855419bf4e00d2a32c452232b324fe910eaadab25 Nov 03 23:53:31.106229 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.106195 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:53:31.106306 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.106265 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:53:31.106306 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.106293 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:53:31.145046 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.145012 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:31.812315 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.812281 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n"] Nov 03 23:53:31.816744 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.816698 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.833796 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.833763 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n"] Nov 03 23:53:31.897575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.897520 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c58cce-5906-460b-81ca-0fe50b8881c1-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.897783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.897610 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrb2x\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-kube-api-access-mrb2x\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.897783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.897677 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.897925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.897850 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.897986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.897951 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.898199 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.898174 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.898282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.898225 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.898361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.898338 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.898425 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.898385 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.985954 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.985916 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" event={"ID":"29e697bb-1d96-4ee6-aa78-78afce02e434","Type":"ContainerStarted","Data":"491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040"} Nov 03 23:53:31.986204 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.985977 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" event={"ID":"29e697bb-1d96-4ee6-aa78-78afce02e434","Type":"ContainerStarted","Data":"9a950dd712ca8f97a509162855419bf4e00d2a32c452232b324fe910eaadab25"} Nov 03 23:53:31.999357 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999322 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999357 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999361 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999572 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999402 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c58cce-5906-460b-81ca-0fe50b8881c1-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999572 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999455 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrb2x\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-kube-api-access-mrb2x\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999572 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999474 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999572 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999563 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999611 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999764 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999873 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999792 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999872 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-credential-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:31.999979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:31.999920 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-certs\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:32.000125 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.000105 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-socket\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:32.000191 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.000156 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-data\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:32.000341 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.000324 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c58cce-5906-460b-81ca-0fe50b8881c1-istiod-ca-cert\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:32.001827 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.001800 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-envoy\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:32.002158 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.002140 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-podinfo\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:32.012556 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.012526 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-token\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:32.012556 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.012546 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrb2x\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-kube-api-access-mrb2x\") pod \"e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:32.013860 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.013825 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" podStartSLOduration=2.013813391 podStartE2EDuration="2.013813391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:32.009296621 +0000 UTC m=+1430.831892784" watchObservedRunningTime="2026-11-03 23:53:32.013813391 +0000 UTC m=+1430.836409553" Nov 03 23:53:32.136355 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.136268 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:32.194133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.194084 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:32.449305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.449277 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n"] Nov 03 23:53:32.449862 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:53:32.449834 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c58cce_5906_460b_81ca_0fe50b8881c1.slice/crio-e3c8cf67deeff8bec0c13b918ec197c2e51c95b57021ff461d8596482697f74b WatchSource:0}: Error finding container e3c8cf67deeff8bec0c13b918ec197c2e51c95b57021ff461d8596482697f74b: Status 404 returned error can't find the container with id e3c8cf67deeff8bec0c13b918ec197c2e51c95b57021ff461d8596482697f74b Nov 03 23:53:32.452450 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.452417 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:53:32.452547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.452482 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:53:32.452547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.452510 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:53:32.480574 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.480533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:32.812659 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.812625 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:32.814895 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.814860 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:53:32.815042 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.814920 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:53:32.994883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.994846 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" event={"ID":"a3c58cce-5906-460b-81ca-0fe50b8881c1","Type":"ContainerStarted","Data":"e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b"} Nov 03 23:53:32.995058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:32.994911 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" event={"ID":"a3c58cce-5906-460b-81ca-0fe50b8881c1","Type":"ContainerStarted","Data":"e3c8cf67deeff8bec0c13b918ec197c2e51c95b57021ff461d8596482697f74b"} Nov 03 23:53:33.023426 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:33.023376 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" podStartSLOduration=2.023361516 podStartE2EDuration="2.023361516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:53:33.019077659 +0000 UTC m=+1431.841673825" watchObservedRunningTime="2026-11-03 23:53:33.023361516 +0000 UTC m=+1431.845957677" Nov 03 23:53:33.137146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:33.137057 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:33.138513 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:33.138488 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:53:33.138689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:33.138671 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:53:33.490057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:33.482592 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:33.812482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:33.812391 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:53:33.812644 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:33.812479 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:53:34.019411 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:34.018365 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:34.133913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:34.133430 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:34.136925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:34.136898 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:53:34.137077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:34.136953 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:53:34.813626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:34.813582 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:53:34.813840 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:34.813638 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:53:34.897057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:34.896944 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:35.137455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:35.137356 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:53:35.137455 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:35.137418 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:53:35.739496 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:53:35.739457 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:53:35.812589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:35.812553 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:53:35.812768 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:35.812609 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:53:35.934068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:35.933214 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:36.045608 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:36.045499 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:36.142554 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:36.142519 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:53:36.143207 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:36.143177 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:53:36.270790 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:36.270753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:36.813304 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:36.813267 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:53:36.813489 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:36.813315 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:53:37.136885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:37.136806 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:53:37.136885 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:37.136871 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:53:37.282056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:37.281011 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:37.813080 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:37.813023 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:53:37.813278 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:37.813129 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:53:38.098309 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:38.098197 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:38.136636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:38.136604 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:53:38.136795 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:38.136660 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:53:38.818288 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:38.818243 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:38.818791 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:38.818716 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:38.820524 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:38.820499 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 03 23:53:38.937492 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:38.937456 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:39.137821 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:39.137732 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:53:39.140109 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:39.140070 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:53:39.703148 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:39.701809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:39.821591 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:39.821552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:40.202110 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:40.202069 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:40.202549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:40.202531 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:40.203257 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:40.203241 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 03 23:53:40.232984 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:40.232945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:41.234834 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:41.234792 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:41.572164 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:41.572067 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:41.676498 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:41.676459 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:42.192486 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:42.192455 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:42.336496 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:42.336461 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:42.840442 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:42.840399 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:44.249177 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:44.249127 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 23:53:44.539313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:44.539223 6208 patch_prober.go:29] interesting pod/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:53:44.539624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:44.539591 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:44.687469 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:44.687412 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:44.728088 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:44.718228 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 03 23:53:44.873771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:44.868082 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:45.274518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.274455 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:45.277297 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.277272 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:53:45.407056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.406426 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:45.567242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.567142 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:45.707530 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.707486 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Nov 03 23:53:45.709838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.709811 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:53:45.729282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.729240 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:45.731682 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.731655 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:53:45.867565 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.867467 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:45.912061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.868408 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:45.912955 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:45.912928 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:53:46.285166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:46.285113 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:53:46.288707 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:46.288661 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:53:46.332180 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:46.329885 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:53:46.335185 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:46.334719 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 03 23:53:46.336661 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:46.336643 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:53:47.549564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:47.549529 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:48.193568 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:48.193535 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:48.293858 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:48.293821 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:48.459308 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:48.459272 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:49.250638 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:49.250601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:50.140729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:50.140693 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:50.944735 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:50.944703 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:51.736130 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:51.736100 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:52.449565 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:52.449507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:53.421603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:53.421570 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:53.523531 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:53.523502 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:54.479306 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:54.479268 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:55.338415 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:55.338379 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:56.191942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:56.191894 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:56.291565 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:56.291533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:57.139368 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:57.139334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:58.056175 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:58.056146 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:58.731413 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:58.731375 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:59.087243 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:59.087166 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t"] Nov 03 23:53:59.087573 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:59.087525 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" containerID="cri-o://a599e8dc7418d9756f8081eea1f2c283b8901e4139f70eb77cf962bb9d7cd5dc" gracePeriod=30 Nov 03 23:53:59.112547 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:59.112514 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:53:59.449920 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:53:59.449825 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:00.294138 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:00.294106 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:01.036098 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:01.036064 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:01.823864 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:01.823827 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:01.949605 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:01.949573 6208 patch_prober.go:29] interesting pod/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:54:01.949785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:01.949628 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:54:02.836986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:02.836944 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:04.022069 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:04.017965 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:04.280116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:04.263591 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:04.542417 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:04.542314 6208 generic.go:356] "Generic (PLEG): container finished" podID="35bd2754-7d47-4cec-830e-45de493c7724" containerID="a599e8dc7418d9756f8081eea1f2c283b8901e4139f70eb77cf962bb9d7cd5dc" exitCode=0 Nov 03 23:54:04.542611 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:04.542519 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" event={"ID":"35bd2754-7d47-4cec-830e-45de493c7724","Type":"ContainerDied","Data":"a599e8dc7418d9756f8081eea1f2c283b8901e4139f70eb77cf962bb9d7cd5dc"} Nov 03 23:54:04.542611 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:04.542582 6208 scope.go:122] "RemoveContainer" containerID="e3867151e611d3a1ca47618c771643f5390a2b39ce8ba97a2f78ebc3b7111e08" Nov 03 23:54:05.313059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.309078 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:05.341015 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.340974 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:05.390479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.390398 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:54:05.509699 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.509646 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-istio-token\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-istio-token\") pod \"35bd2754-7d47-4cec-830e-45de493c7724\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " Nov 03 23:54:05.511456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.511427 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bd2754-7d47-4cec-830e-45de493c7724-istiod-ca-cert" pod "35bd2754-7d47-4cec-830e-45de493c7724" (UID: "35bd2754-7d47-4cec-830e-45de493c7724"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:54:05.511648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.511628 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35bd2754-7d47-4cec-830e-45de493c7724-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bd2754-7d47-4cec-830e-45de493c7724-istiod-ca-cert\") pod \"35bd2754-7d47-4cec-830e-45de493c7724\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " Nov 03 23:54:05.512166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.512145 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-data\") pod \"35bd2754-7d47-4cec-830e-45de493c7724\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " Nov 03 23:54:05.512352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.512334 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-credential-socket\") pod \"35bd2754-7d47-4cec-830e-45de493c7724\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " Nov 03 23:54:05.513131 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.512789 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-credential-socket" pod "35bd2754-7d47-4cec-830e-45de493c7724" (UID: "35bd2754-7d47-4cec-830e-45de493c7724"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:05.513282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.512961 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-data" pod "35bd2754-7d47-4cec-830e-45de493c7724" (UID: "35bd2754-7d47-4cec-830e-45de493c7724"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:05.513761 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.513742 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35bd2754-7d47-4cec-830e-45de493c7724-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bd2754-7d47-4cec-830e-45de493c7724-istio-podinfo\") pod \"35bd2754-7d47-4cec-830e-45de493c7724\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " Nov 03 23:54:05.514180 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.514151 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-socket\") pod \"35bd2754-7d47-4cec-830e-45de493c7724\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " Nov 03 23:54:05.514583 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.514561 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-envoy\") pod \"35bd2754-7d47-4cec-830e-45de493c7724\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " Nov 03 23:54:05.515134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.515116 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-certs\") pod \"35bd2754-7d47-4cec-830e-45de493c7724\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " Nov 03 23:54:05.515517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.515500 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-kube-api-access-gt5dz\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-kube-api-access-gt5dz\") pod \"35bd2754-7d47-4cec-830e-45de493c7724\" (UID: \"35bd2754-7d47-4cec-830e-45de493c7724\") " Nov 03 23:54:05.516685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.516664 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bd2754-7d47-4cec-830e-45de493c7724-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:05.516832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.516818 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:05.516963 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.516950 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:05.517136 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.515013 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-socket" pod "35bd2754-7d47-4cec-830e-45de493c7724" (UID: "35bd2754-7d47-4cec-830e-45de493c7724"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:05.517244 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.515450 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-certs" pod "35bd2754-7d47-4cec-830e-45de493c7724" (UID: "35bd2754-7d47-4cec-830e-45de493c7724"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:05.530431 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.529272 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35bd2754-7d47-4cec-830e-45de493c7724-istio-podinfo" pod "35bd2754-7d47-4cec-830e-45de493c7724" (UID: "35bd2754-7d47-4cec-830e-45de493c7724"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:54:05.533991 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.533484 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-envoy" pod "35bd2754-7d47-4cec-830e-45de493c7724" (UID: "35bd2754-7d47-4cec-830e-45de493c7724"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:05.533991 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.533570 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-kube-api-access-gt5dz" pod "35bd2754-7d47-4cec-830e-45de493c7724" (UID: "35bd2754-7d47-4cec-830e-45de493c7724"). InnerVolumeSpecName "kube-api-access-gt5dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:54:05.535321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.535296 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-istio-token" pod "35bd2754-7d47-4cec-830e-45de493c7724" (UID: "35bd2754-7d47-4cec-830e-45de493c7724"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:54:05.551616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.551536 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" event={"ID":"35bd2754-7d47-4cec-830e-45de493c7724","Type":"ContainerDied","Data":"4ba716c1953d15749846f5cbecfdc6e551983054dbabefe632f3caf967f53c9f"} Nov 03 23:54:05.551616 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.551598 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t" Nov 03 23:54:05.552016 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.551606 6208 scope.go:122] "RemoveContainer" containerID="a599e8dc7418d9756f8081eea1f2c283b8901e4139f70eb77cf962bb9d7cd5dc" Nov 03 23:54:05.575473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.575443 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:05.608829 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.608787 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t"] Nov 03 23:54:05.620724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.620642 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:05.620970 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.620952 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bd2754-7d47-4cec-830e-45de493c7724-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:05.621087 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.621075 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:05.621179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.621168 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:05.621266 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.621255 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bd2754-7d47-4cec-830e-45de493c7724-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:05.621347 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.621336 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5dz\" (UniqueName: \"kubernetes.io/projected/35bd2754-7d47-4cec-830e-45de493c7724-kube-api-access-gt5dz\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:05.636223 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:05.636186 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c7e425-openshift-default-6f6b568549-7lx5t"] Nov 03 23:54:05.733144 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:54:05.733105 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:54:06.117880 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:06.117801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:06.218726 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:06.218689 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:06.628987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:06.628952 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:07.296271 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:07.296239 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35bd2754-7d47-4cec-830e-45de493c7724" path="/var/lib/kubelet/pods/35bd2754-7d47-4cec-830e-45de493c7724/volumes" Nov 03 23:54:07.518555 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:07.518518 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:08.439219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:08.439175 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:09.174067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:09.173324 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:10.054446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:10.054407 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:10.869373 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:10.869338 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:11.640189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:11.640153 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:12.290434 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:12.290390 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:12.388421 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:12.388389 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:12.488187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:12.488152 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:13.245456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:13.245416 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:13.513359 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:13.513270 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:13.636914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:13.636885 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:14.040414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:14.040382 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:14.948286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:14.948246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:15.048578 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:15.048539 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:15.848328 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:15.848293 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:16.468865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:16.468816 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:16.936598 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:16.936490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:17.922556 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:17.922520 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:18.741305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:18.741272 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:19.537833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:19.537798 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:20.254300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:20.254264 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:20.360734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:20.360704 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:21.259498 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:21.259467 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:22.020520 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:22.020487 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:22.453241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:22.453153 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8"] Nov 03 23:54:22.461968 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:22.461933 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9"] Nov 03 23:54:22.494708 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:22.494662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:22.595079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:22.595011 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:23.023255 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:23.023201 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:23.839201 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:23.839169 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:24.550752 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:24.550713 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:26.139669 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:26.139635 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:26.345079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:26.344146 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:26.533059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:26.532170 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:26.563815 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:26.563160 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:26.564576 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:26.564545 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Nov 03 23:54:26.891223 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:26.890980 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:27.454001 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.453963 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" podUID="28e366c5-9003-4bce-bea7-5c131ada5520" containerName="payload-processing" containerID="cri-o://101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c" gracePeriod=25 Nov 03 23:54:27.462593 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.462559 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" podUID="f004907f-237a-4f81-aa9d-e8d28ddb76cf" containerName="payload-pre-processing" containerID="cri-o://443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e" gracePeriod=25 Nov 03 23:54:27.757386 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.757356 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:54:27.764742 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.764709 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e366c5-9003-4bce-bea7-5c131ada5520-kube-api-access-c9rfs\" (UniqueName: \"kubernetes.io/projected/28e366c5-9003-4bce-bea7-5c131ada5520-kube-api-access-c9rfs\") pod \"28e366c5-9003-4bce-bea7-5c131ada5520\" (UID: \"28e366c5-9003-4bce-bea7-5c131ada5520\") " Nov 03 23:54:27.765072 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.765019 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e366c5-9003-4bce-bea7-5c131ada5520-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e366c5-9003-4bce-bea7-5c131ada5520-plugins-config-volume\") pod \"28e366c5-9003-4bce-bea7-5c131ada5520\" (UID: \"28e366c5-9003-4bce-bea7-5c131ada5520\") " Nov 03 23:54:27.765968 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.765947 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e366c5-9003-4bce-bea7-5c131ada5520-plugins-config-volume" pod "28e366c5-9003-4bce-bea7-5c131ada5520" (UID: "28e366c5-9003-4bce-bea7-5c131ada5520"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:54:27.767596 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.767567 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e366c5-9003-4bce-bea7-5c131ada5520-kube-api-access-c9rfs" pod "28e366c5-9003-4bce-bea7-5c131ada5520" (UID: "28e366c5-9003-4bce-bea7-5c131ada5520"). InnerVolumeSpecName "kube-api-access-c9rfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:54:27.803198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.803166 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:54:27.866718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.866676 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f004907f-237a-4f81-aa9d-e8d28ddb76cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f004907f-237a-4f81-aa9d-e8d28ddb76cf-plugins-config-volume\") pod \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\" (UID: \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\") " Nov 03 23:54:27.866910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.866862 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f004907f-237a-4f81-aa9d-e8d28ddb76cf-kube-api-access-9fr58\" (UniqueName: \"kubernetes.io/projected/f004907f-237a-4f81-aa9d-e8d28ddb76cf-kube-api-access-9fr58\") pod \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\" (UID: \"f004907f-237a-4f81-aa9d-e8d28ddb76cf\") " Nov 03 23:54:27.867157 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.867126 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f004907f-237a-4f81-aa9d-e8d28ddb76cf-plugins-config-volume" pod "f004907f-237a-4f81-aa9d-e8d28ddb76cf" (UID: "f004907f-237a-4f81-aa9d-e8d28ddb76cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:54:27.867489 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.867470 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9rfs\" (UniqueName: \"kubernetes.io/projected/28e366c5-9003-4bce-bea7-5c131ada5520-kube-api-access-c9rfs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:27.867560 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.867490 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e366c5-9003-4bce-bea7-5c131ada5520-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:27.867560 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.867502 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f004907f-237a-4f81-aa9d-e8d28ddb76cf-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:27.869556 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.869530 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f004907f-237a-4f81-aa9d-e8d28ddb76cf-kube-api-access-9fr58" pod "f004907f-237a-4f81-aa9d-e8d28ddb76cf" (UID: "f004907f-237a-4f81-aa9d-e8d28ddb76cf"). InnerVolumeSpecName "kube-api-access-9fr58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:54:27.921317 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.921282 6208 generic.go:356] "Generic (PLEG): container finished" podID="28e366c5-9003-4bce-bea7-5c131ada5520" containerID="101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c" exitCode=0 Nov 03 23:54:27.921510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.921334 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" Nov 03 23:54:27.921510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.921362 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" event={"ID":"28e366c5-9003-4bce-bea7-5c131ada5520","Type":"ContainerDied","Data":"101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c"} Nov 03 23:54:27.921510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.921433 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8" event={"ID":"28e366c5-9003-4bce-bea7-5c131ada5520","Type":"ContainerDied","Data":"4236dbce6cdb2682d901619f2439c82e37ee1db1af95f6d6f70920c5184d9dd0"} Nov 03 23:54:27.921510 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.921467 6208 scope.go:122] "RemoveContainer" containerID="101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c" Nov 03 23:54:27.923507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.923478 6208 generic.go:356] "Generic (PLEG): container finished" podID="f004907f-237a-4f81-aa9d-e8d28ddb76cf" containerID="443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e" exitCode=0 Nov 03 23:54:27.923595 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.923520 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" Nov 03 23:54:27.923595 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.923561 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" event={"ID":"f004907f-237a-4f81-aa9d-e8d28ddb76cf","Type":"ContainerDied","Data":"443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e"} Nov 03 23:54:27.923670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.923606 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9" event={"ID":"f004907f-237a-4f81-aa9d-e8d28ddb76cf","Type":"ContainerDied","Data":"a656d6f3bd27679672f75ccb8749d2ae224915ddc49f5f7c8a6d6d4e9f8a3fb5"} Nov 03 23:54:27.934225 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.934203 6208 scope.go:122] "RemoveContainer" containerID="101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c" Nov 03 23:54:27.934627 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:54:27.934597 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c\": container with ID starting with 101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c not found: ID does not exist" containerID="101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c" Nov 03 23:54:27.934753 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.934637 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c"} err="failed to get container status \"101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c\": rpc error: code = NotFound desc = could not find container \"101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c\": container with ID starting with 101856f4637fcb726d7eb78cabbab04a4e667ce2224da6e64d02e16ab097c40c not found: ID does not exist" Nov 03 23:54:27.934753 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.934655 6208 scope.go:122] "RemoveContainer" containerID="443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e" Nov 03 23:54:27.948167 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.948146 6208 scope.go:122] "RemoveContainer" containerID="443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e" Nov 03 23:54:27.948728 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:54:27.948648 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e\": container with ID starting with 443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e not found: ID does not exist" containerID="443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e" Nov 03 23:54:27.948728 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.948685 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e"} err="failed to get container status \"443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e\": rpc error: code = NotFound desc = could not find container \"443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e\": container with ID starting with 443918cfb2b08a636859a7cdf29e10c0fc2ecfbaa24a6323fbf8a1bee07efd7e not found: ID does not exist" Nov 03 23:54:27.961178 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.961108 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8"] Nov 03 23:54:27.965730 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.965702 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-d6e155-787bd49586-jphw8"] Nov 03 23:54:27.969043 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.969003 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fr58\" (UniqueName: \"kubernetes.io/projected/f004907f-237a-4f81-aa9d-e8d28ddb76cf-kube-api-access-9fr58\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:27.971451 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.971429 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9"] Nov 03 23:54:27.975236 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.975211 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:27.977647 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:27.977617 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d6e155-7cfcf95495-skxm9"] Nov 03 23:54:28.928600 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:28.928498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:29.296341 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.296307 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e366c5-9003-4bce-bea7-5c131ada5520" path="/var/lib/kubelet/pods/28e366c5-9003-4bce-bea7-5c131ada5520/volumes" Nov 03 23:54:29.296735 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.296718 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f004907f-237a-4f81-aa9d-e8d28ddb76cf" path="/var/lib/kubelet/pods/f004907f-237a-4f81-aa9d-e8d28ddb76cf/volumes" Nov 03 23:54:29.530975 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.530936 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj"] Nov 03 23:54:29.531223 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.531194 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" podUID="5b605151-325f-4bd0-b11d-23a2225bc799" containerName="maas-api" containerID="cri-o://615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48" gracePeriod=30 Nov 03 23:54:29.571947 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.571918 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:29.856014 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.855649 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:29.918969 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.918934 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:54:29.943027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.942995 6208 generic.go:356] "Generic (PLEG): container finished" podID="5b605151-325f-4bd0-b11d-23a2225bc799" containerID="615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48" exitCode=0 Nov 03 23:54:29.943535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.943218 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" event={"ID":"5b605151-325f-4bd0-b11d-23a2225bc799","Type":"ContainerDied","Data":"615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48"} Nov 03 23:54:29.943535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.943269 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" event={"ID":"5b605151-325f-4bd0-b11d-23a2225bc799","Type":"ContainerDied","Data":"1d216de375d94906be0355d55126c94cbd90b55dba67c93077fde47982b6a2fe"} Nov 03 23:54:29.943535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.943303 6208 scope.go:122] "RemoveContainer" containerID="615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48" Nov 03 23:54:29.943535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.943396 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj" Nov 03 23:54:29.957492 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.957466 6208 scope.go:122] "RemoveContainer" containerID="615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48" Nov 03 23:54:29.958249 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:54:29.958219 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48\": container with ID starting with 615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48 not found: ID does not exist" containerID="615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48" Nov 03 23:54:29.958499 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.958472 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48"} err="failed to get container status \"615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48\": rpc error: code = NotFound desc = could not find container \"615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48\": container with ID starting with 615d1af45848341cf1c47c0d28041a94577716f79f7bf9a3faef6d70a73b7a48 not found: ID does not exist" Nov 03 23:54:29.998847 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:29.998811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:30.007083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.007051 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b605151-325f-4bd0-b11d-23a2225bc799-kube-api-access-9pdvr\" (UniqueName: \"kubernetes.io/projected/5b605151-325f-4bd0-b11d-23a2225bc799-kube-api-access-9pdvr\") pod \"5b605151-325f-4bd0-b11d-23a2225bc799\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " Nov 03 23:54:30.007248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.007214 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-maas-api-tls\") pod \"5b605151-325f-4bd0-b11d-23a2225bc799\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " Nov 03 23:54:30.007305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.007264 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-metrics-certs\") pod \"5b605151-325f-4bd0-b11d-23a2225bc799\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " Nov 03 23:54:30.007401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.007386 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b605151-325f-4bd0-b11d-23a2225bc799-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b605151-325f-4bd0-b11d-23a2225bc799-openshift-service-ca\") pod \"5b605151-325f-4bd0-b11d-23a2225bc799\" (UID: \"5b605151-325f-4bd0-b11d-23a2225bc799\") " Nov 03 23:54:30.007755 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.007725 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b605151-325f-4bd0-b11d-23a2225bc799-openshift-service-ca" pod "5b605151-325f-4bd0-b11d-23a2225bc799" (UID: "5b605151-325f-4bd0-b11d-23a2225bc799"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:54:30.007927 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.007907 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b605151-325f-4bd0-b11d-23a2225bc799-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:30.009246 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.009225 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b605151-325f-4bd0-b11d-23a2225bc799-kube-api-access-9pdvr" pod "5b605151-325f-4bd0-b11d-23a2225bc799" (UID: "5b605151-325f-4bd0-b11d-23a2225bc799"). InnerVolumeSpecName "kube-api-access-9pdvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:54:30.009721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.009699 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-metrics-certs" pod "5b605151-325f-4bd0-b11d-23a2225bc799" (UID: "5b605151-325f-4bd0-b11d-23a2225bc799"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:54:30.009794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.009724 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-maas-api-tls" pod "5b605151-325f-4bd0-b11d-23a2225bc799" (UID: "5b605151-325f-4bd0-b11d-23a2225bc799"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:54:30.109625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.109545 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pdvr\" (UniqueName: \"kubernetes.io/projected/5b605151-325f-4bd0-b11d-23a2225bc799-kube-api-access-9pdvr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:30.109625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.109577 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:30.109625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.109588 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b605151-325f-4bd0-b11d-23a2225bc799-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:30.276272 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.276237 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj"] Nov 03 23:54:30.279427 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.279398 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-d6e155-74c9d9d8cb-wkrfj"] Nov 03 23:54:30.371535 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.371435 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:30.783304 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.783266 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:30.930060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:30.930008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:31.209882 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:31.209852 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:31.295559 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:31.295528 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b605151-325f-4bd0-b11d-23a2225bc799" path="/var/lib/kubelet/pods/5b605151-325f-4bd0-b11d-23a2225bc799/volumes" Nov 03 23:54:31.310819 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:31.310791 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:32.033174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:32.033142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:32.449725 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:32.449596 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:33.482118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:33.482061 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:33.874350 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:33.874261 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:33.976596 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:33.976551 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:34.532613 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:34.532564 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:35.228680 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:35.228648 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:35.343830 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:35.343799 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:35.704845 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:54:35.704768 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:54:35.918401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:35.918367 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:36.829113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:36.829077 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:37.092267 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:37.092157 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:37.192549 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:37.192510 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:37.635466 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:37.635438 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:37.922250 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:37.922168 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:37.996295 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:37.996263 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k"] Nov 03 23:54:37.996711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:37.996682 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" containerID="cri-o://ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660" gracePeriod=30 Nov 03 23:54:38.027106 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:38.027070 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:38.812401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:38.812363 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:39.199443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:39.199368 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:39.308626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:39.308566 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:39.969483 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:39.969434 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:40.362768 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:40.362682 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:40.910775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:40.910727 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:41.011865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:41.011830 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:41.627252 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:41.627209 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:42.103690 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:42.103653 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:42.177197 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:42.177171 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:54:42.212482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:42.212446 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:54:42.427883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:42.427802 6208 patch_prober.go:29] interesting pod/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:54:42.427883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:42.427854 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:54:42.923795 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:42.923760 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:43.279125 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.279100 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:54:43.415938 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.415897 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-certs\") pod \"5ad6532c-46b9-439c-88d3-395fe0b78a50\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " Nov 03 23:54:43.416179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.415983 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-kube-api-access-7bcfb\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-kube-api-access-7bcfb\") pod \"5ad6532c-46b9-439c-88d3-395fe0b78a50\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " Nov 03 23:54:43.416179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416083 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-credential-socket\") pod \"5ad6532c-46b9-439c-88d3-395fe0b78a50\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " Nov 03 23:54:43.416179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416109 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-token\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-token\") pod \"5ad6532c-46b9-439c-88d3-395fe0b78a50\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " Nov 03 23:54:43.416179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416133 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-data\") pod \"5ad6532c-46b9-439c-88d3-395fe0b78a50\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " Nov 03 23:54:43.416179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416157 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-certs" pod "5ad6532c-46b9-439c-88d3-395fe0b78a50" (UID: "5ad6532c-46b9-439c-88d3-395fe0b78a50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:43.416454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416196 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-envoy\") pod \"5ad6532c-46b9-439c-88d3-395fe0b78a50\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " Nov 03 23:54:43.416454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416243 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad6532c-46b9-439c-88d3-395fe0b78a50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad6532c-46b9-439c-88d3-395fe0b78a50-istiod-ca-cert\") pod \"5ad6532c-46b9-439c-88d3-395fe0b78a50\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " Nov 03 23:54:43.416454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416289 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-socket\") pod \"5ad6532c-46b9-439c-88d3-395fe0b78a50\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " Nov 03 23:54:43.416454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416405 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-credential-socket" pod "5ad6532c-46b9-439c-88d3-395fe0b78a50" (UID: "5ad6532c-46b9-439c-88d3-395fe0b78a50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:43.416454 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416417 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-data" pod "5ad6532c-46b9-439c-88d3-395fe0b78a50" (UID: "5ad6532c-46b9-439c-88d3-395fe0b78a50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:43.416705 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416616 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-socket" pod "5ad6532c-46b9-439c-88d3-395fe0b78a50" (UID: "5ad6532c-46b9-439c-88d3-395fe0b78a50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:43.416705 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416649 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad6532c-46b9-439c-88d3-395fe0b78a50-istiod-ca-cert" pod "5ad6532c-46b9-439c-88d3-395fe0b78a50" (UID: "5ad6532c-46b9-439c-88d3-395fe0b78a50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:54:43.416705 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.416654 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-podinfo\") pod \"5ad6532c-46b9-439c-88d3-395fe0b78a50\" (UID: \"5ad6532c-46b9-439c-88d3-395fe0b78a50\") " Nov 03 23:54:43.417562 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.417536 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:43.417693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.417565 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:43.417693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.417582 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:43.417693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.417596 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad6532c-46b9-439c-88d3-395fe0b78a50-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:43.417693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.417609 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:43.418482 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.418459 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-token" pod "5ad6532c-46b9-439c-88d3-395fe0b78a50" (UID: "5ad6532c-46b9-439c-88d3-395fe0b78a50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:54:43.418575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.418469 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-envoy" pod "5ad6532c-46b9-439c-88d3-395fe0b78a50" (UID: "5ad6532c-46b9-439c-88d3-395fe0b78a50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:54:43.418575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.418486 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-podinfo" pod "5ad6532c-46b9-439c-88d3-395fe0b78a50" (UID: "5ad6532c-46b9-439c-88d3-395fe0b78a50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:54:43.418575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.418492 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-kube-api-access-7bcfb" pod "5ad6532c-46b9-439c-88d3-395fe0b78a50" (UID: "5ad6532c-46b9-439c-88d3-395fe0b78a50"). InnerVolumeSpecName "kube-api-access-7bcfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:54:43.518832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.518744 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:43.518832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.518778 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:43.518832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.518788 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bcfb\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-kube-api-access-7bcfb\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:43.518832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.518799 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad6532c-46b9-439c-88d3-395fe0b78a50-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:54:43.645339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:43.645300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:44.099660 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.099626 6208 generic.go:356] "Generic (PLEG): container finished" podID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerID="ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660" exitCode=0 Nov 03 23:54:44.100135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.099691 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" Nov 03 23:54:44.100135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.099713 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" event={"ID":"5ad6532c-46b9-439c-88d3-395fe0b78a50","Type":"ContainerDied","Data":"ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660"} Nov 03 23:54:44.100135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.099797 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k" event={"ID":"5ad6532c-46b9-439c-88d3-395fe0b78a50","Type":"ContainerDied","Data":"d26827c58d3a767f008beb7c1da93510a4890bb11448d5025db187e98698e39e"} Nov 03 23:54:44.100135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.099821 6208 scope.go:122] "RemoveContainer" containerID="ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660" Nov 03 23:54:44.111567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.111550 6208 scope.go:122] "RemoveContainer" containerID="8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf" Nov 03 23:54:44.121804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.121787 6208 scope.go:122] "RemoveContainer" containerID="ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660" Nov 03 23:54:44.122120 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:54:44.122100 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660\": container with ID starting with ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660 not found: ID does not exist" containerID="ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660" Nov 03 23:54:44.122196 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.122127 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660"} err="failed to get container status \"ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660\": rpc error: code = NotFound desc = could not find container \"ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660\": container with ID starting with ed16bbe5a6fa60aa8ed9447b6e4d68c2a55705f9b33cc4fb46c2932a5ba64660 not found: ID does not exist" Nov 03 23:54:44.122196 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.122142 6208 scope.go:122] "RemoveContainer" containerID="8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf" Nov 03 23:54:44.122364 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:54:44.122351 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf\": container with ID starting with 8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf not found: ID does not exist" containerID="8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf" Nov 03 23:54:44.122411 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.122368 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf"} err="failed to get container status \"8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf\": rpc error: code = NotFound desc = could not find container \"8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf\": container with ID starting with 8627eae5e34d79f3a6c73f4d29f3277aad83c73bfe00503e76175b6c13d4e2bf not found: ID does not exist" Nov 03 23:54:44.137642 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.137617 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k"] Nov 03 23:54:44.142968 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.142944 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-d6e155-openshift-default-758cbbf8b5-ctw9k"] Nov 03 23:54:44.514247 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:44.514212 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:45.266568 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:45.266536 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:45.296974 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:45.296945 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" path="/var/lib/kubelet/pods/5ad6532c-46b9-439c-88d3-395fe0b78a50/volumes" Nov 03 23:54:46.076083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:46.076047 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:46.856634 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:46.856588 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:47.357939 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:47.357904 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:47.747234 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:47.747187 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:47.861925 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:47.861876 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:48.008809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:48.008729 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:48.550497 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:48.550467 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:48.753464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:48.753425 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:49.604155 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:49.604107 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:49.760173 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:49.760121 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:50.208344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.208106 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:50.294057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.290753 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 23:54:50.298132 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.294027 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 23:54:50.367231 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.367179 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:50.393269 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.393219 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:50.609687 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.609554 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:50.610897 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.610859 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:54:50.764591 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.764540 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:50.769602 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.769573 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:54:50.841059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.820140 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:54:50.841059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:50.820213 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:51.040703 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.025727 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:51.141581 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.141545 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:51.402132 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.382308 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:54:51.420286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.411865 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 23:54:51.420543 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.411932 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 03 23:54:51.642889 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.642839 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 03 23:54:51.671285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.670694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:51.738351 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.738323 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:54:51.738852 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.738835 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:54:51.739011 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.739001 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:54:51.739147 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.739137 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 03 23:54:51.814081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.790583 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 03 23:54:51.814081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.813268 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 03 23:54:51.814081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.813315 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 03 23:54:51.814081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:51.813348 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 03 23:54:53.773457 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:53.773422 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:54.654531 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:54.654497 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:55.684475 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:55.684440 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:55.932189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:55.932149 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:56.070732 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:56.070693 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:56.724695 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:56.724645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:57.128861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:57.128775 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:57.242747 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:57.242712 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:57.720023 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:57.719973 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:58.801418 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:58.801372 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:59.126100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:59.125991 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:54:59.937587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:54:59.937535 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:00.744607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:00.744578 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:01.537726 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:01.537693 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:02.427437 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:02.427399 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:02.530448 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:02.530403 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:02.631381 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:02.631344 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:03.352493 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:03.352412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:04.211375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:04.211340 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:05.128240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:05.128211 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:05.712389 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:55:05.712350 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:55:06.037096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:06.037010 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:06.240568 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:06.240534 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:06.656529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:06.656496 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:06.845587 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:06.845555 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:07.113450 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:07.113412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:07.524868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:07.524837 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:07.879243 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:07.879160 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:08.640440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:08.640405 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:09.537174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:09.537130 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:09.642575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:09.642529 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:10.122670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:10.122630 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:10.232989 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:10.232920 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:11.438552 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:11.438518 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:11.625058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:11.620232 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:12.470058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:12.469467 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:12.518055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:12.510343 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:55:12.678146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:12.677669 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:55:12.744186 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:12.743358 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:55:12.786212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:12.786160 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:55:12.786534 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:12.786481 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:55:12.799509 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:12.798189 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:55:12.799509 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:12.798862 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:55:12.810238 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:12.810209 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:55:13.410850 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:13.410808 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:13.514336 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:13.514304 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:14.835951 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:14.835920 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:14.952537 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:14.952493 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:15.054748 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:15.054712 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:16.073083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.073024 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:16.207078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.207025 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:16.320420 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.320387 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:16.513845 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.513816 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:16.698782 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.698729 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:16.929379 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.929305 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6"] Nov 03 23:55:16.929987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.929966 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" Nov 03 23:55:16.929987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.929986 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930001 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5b605151-325f-4bd0-b11d-23a2225bc799" containerName="maas-api" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930006 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b605151-325f-4bd0-b11d-23a2225bc799" containerName="maas-api" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930024 6208 cpu_manager.go:414] "containerMap: removing container" podUID="28e366c5-9003-4bce-bea7-5c131ada5520" containerName="payload-processing" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930044 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e366c5-9003-4bce-bea7-5c131ada5520" containerName="payload-processing" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930068 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f004907f-237a-4f81-aa9d-e8d28ddb76cf" containerName="payload-pre-processing" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930075 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f004907f-237a-4f81-aa9d-e8d28ddb76cf" containerName="payload-pre-processing" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930090 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930102 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930115 6208 cpu_manager.go:414] "containerMap: removing container" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" Nov 03 23:55:16.930161 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930120 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" Nov 03 23:55:16.930471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930223 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" Nov 03 23:55:16.930471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930235 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" Nov 03 23:55:16.930471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930242 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b605151-325f-4bd0-b11d-23a2225bc799" containerName="maas-api" Nov 03 23:55:16.930471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930250 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e366c5-9003-4bce-bea7-5c131ada5520" containerName="payload-processing" Nov 03 23:55:16.930471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.930257 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f004907f-237a-4f81-aa9d-e8d28ddb76cf" containerName="payload-pre-processing" Nov 03 23:55:16.933953 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.933934 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:16.957755 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.957733 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9a5948-openshift-default-dockercfg-28jns\"" Nov 03 23:55:16.960083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.960060 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6"] Nov 03 23:55:16.982624 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:16.982599 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:17.000637 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.000575 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257b95e5-b863-4444-9a35-efd3ca15e157-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.001902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.000758 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.001902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.000819 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vblj4\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-kube-api-access-vblj4\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.001902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.000850 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.001902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.000882 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257b95e5-b863-4444-9a35-efd3ca15e157-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.001902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.001152 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.001902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.001280 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.001902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.001367 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.001902 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.001415 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.107833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.107569 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.107833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.107620 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.107833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.107690 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vblj4\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-kube-api-access-vblj4\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.107833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.107722 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.107833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.107755 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257b95e5-b863-4444-9a35-efd3ca15e157-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.107833 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.107861 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.108468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.107983 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.108468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.108076 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.108468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.108129 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.108468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.108176 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257b95e5-b863-4444-9a35-efd3ca15e157-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.109416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.109391 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.109603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.109583 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.109995 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.109971 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257b95e5-b863-4444-9a35-efd3ca15e157-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.109995 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.109986 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.111752 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.111732 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257b95e5-b863-4444-9a35-efd3ca15e157-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.112056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.112014 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.124486 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.124455 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.124633 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.124512 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vblj4\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-kube-api-access-vblj4\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.238334 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.238229 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:17.247610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.247563 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:17.352816 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.352773 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:17.540682 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.540647 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6"] Nov 03 23:55:17.546335 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.546291 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:17.546452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.546385 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:17.546452 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.546430 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:17.712544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.712505 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" event={"ID":"257b95e5-b863-4444-9a35-efd3ca15e157","Type":"ContainerStarted","Data":"6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e"} Nov 03 23:55:17.712714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.712575 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" event={"ID":"257b95e5-b863-4444-9a35-efd3ca15e157","Type":"ContainerStarted","Data":"9df4c97f299bce5f2e1771b1b4ac681f9fd520f5f880fccc2b2477a9932ec3e8"} Nov 03 23:55:17.742646 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.742583 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" podStartSLOduration=1.742564413 podStartE2EDuration="1.742564413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:17.738417019 +0000 UTC m=+1536.561013181" watchObservedRunningTime="2026-11-03 23:55:17.742564413 +0000 UTC m=+1536.565160577" Nov 03 23:55:17.945125 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:17.944989 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:18.048675 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:18.048638 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:18.154921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:18.154882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:18.247980 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:18.247951 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:18.254409 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:18.254379 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:18.719826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:18.719796 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:18.720670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:18.720651 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:55:19.005647 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:19.005559 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:19.926361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:19.926273 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:20.741585 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:20.741551 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:21.456799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:21.456761 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:21.552629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:21.552597 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:21.653167 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:21.653132 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:21.984149 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:21.984019 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:22.363181 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.363096 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:22.399479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.399442 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s"] Nov 03 23:55:22.400354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.400330 6208 cpu_manager.go:414] "containerMap: removing container" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" Nov 03 23:55:22.400354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.400351 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" Nov 03 23:55:22.400545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.400465 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bd2754-7d47-4cec-830e-45de493c7724" containerName="istio-proxy" Nov 03 23:55:22.400545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.400476 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad6532c-46b9-439c-88d3-395fe0b78a50" containerName="istio-proxy" Nov 03 23:55:22.404361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.404340 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.411441 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.411419 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2341ca-openshift-default-dockercfg-s4vcn\"" Nov 03 23:55:22.418200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.418177 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s"] Nov 03 23:55:22.473083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.473022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:22.486150 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.486117 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-data\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.486319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.486168 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.486319 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.486266 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-podinfo\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.486414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.486371 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-credential-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.486414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.486402 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-token\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.486544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.486517 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nt7f\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-kube-api-access-2nt7f\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.486656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.486568 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-envoy\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.486712 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.486697 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-certs\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.486758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.486732 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istiod-ca-cert\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.576055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.575460 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:22.591055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.590905 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nt7f\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-kube-api-access-2nt7f\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.591055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.590982 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-envoy\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.591307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.591143 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-certs\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.591307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.591185 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istiod-ca-cert\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.591307 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.591276 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-data\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.591476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.591306 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.591476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.591374 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-podinfo\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.591582 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.591508 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-credential-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.591582 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.591553 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-token\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.614739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.595124 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-envoy\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.614739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.595393 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-certs\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.614739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.595990 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istiod-ca-cert\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.614739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.596263 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-data\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.614739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.596521 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.614739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.600391 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-credential-socket\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.614739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.603541 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-podinfo\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.614739 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.606388 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-token\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.627492 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.617778 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nt7f\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-kube-api-access-2nt7f\") pod \"e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:22.720413 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:22.720373 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:23.026209 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.026178 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s"] Nov 03 23:55:23.026971 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:55:23.026944 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7485bb_2460_4c40_8815_bcfaf0a2c9a0.slice/crio-595771f70c81b685826f95e7ddde95b4ae34d15be334d4e01892d0d5dade9d72 WatchSource:0}: Error finding container 595771f70c81b685826f95e7ddde95b4ae34d15be334d4e01892d0d5dade9d72: Status 404 returned error can't find the container with id 595771f70c81b685826f95e7ddde95b4ae34d15be334d4e01892d0d5dade9d72 Nov 03 23:55:23.029577 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.029541 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:23.029663 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.029612 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:23.029663 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.029643 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:23.248341 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.248304 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:23.355120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.353928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:23.532887 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.532855 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:23.761091 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.761052 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" event={"ID":"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0","Type":"ContainerStarted","Data":"30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081"} Nov 03 23:55:23.761266 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.761123 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" event={"ID":"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0","Type":"ContainerStarted","Data":"595771f70c81b685826f95e7ddde95b4ae34d15be334d4e01892d0d5dade9d72"} Nov 03 23:55:23.794045 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:23.793981 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" podStartSLOduration=1.793968601 podStartE2EDuration="1.793968601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:23.789545628 +0000 UTC m=+1542.612141796" watchObservedRunningTime="2026-11-03 23:55:23.793968601 +0000 UTC m=+1542.616564763" Nov 03 23:55:24.342101 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:24.342058 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:24.721338 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:24.721308 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:24.727477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:24.727451 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:24.768361 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:24.768332 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:24.769148 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:24.769131 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:55:25.343149 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:25.343116 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:26.230077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:26.230018 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:26.339121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:26.339085 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:26.933049 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:26.933006 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:27.101456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.101414 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg"] Nov 03 23:55:27.106734 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.106709 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.120800 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.120775 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-b3bea7-openshift-default-dockercfg-x9m4w\"" Nov 03 23:55:27.122073 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.122046 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg"] Nov 03 23:55:27.143520 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.143488 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:27.251540 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.251506 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:27.275054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.273984 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-credential-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.275054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.274051 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-certs\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.275054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.274093 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istiod-ca-cert\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.282054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.281939 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-data\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.282228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.282075 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-envoy\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.282228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.282153 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2mh\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-kube-api-access-wh2mh\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.282350 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.282254 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.282350 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.282286 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-token\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.282350 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.282327 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-podinfo\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384103 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-credential-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384331 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384148 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-certs\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384331 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384181 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istiod-ca-cert\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384331 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384201 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-data\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384331 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384229 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-envoy\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384331 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384299 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2mh\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-kube-api-access-wh2mh\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384416 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384449 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-token\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384500 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-podinfo\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384550 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-credential-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384598 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-certs\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384817 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-socket\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.384960 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384934 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-data\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.385174 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.384981 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istiod-ca-cert\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.387189 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.387160 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-envoy\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.387571 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.387554 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-podinfo\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.397241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.397220 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2mh\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-kube-api-access-wh2mh\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.397575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.397558 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-token\") pod \"e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.428421 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.428395 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:27.510297 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.510217 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:27.625642 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.625604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:27.758061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.751591 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:27.839188 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:55:27.818405 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7240cb0b_5340_4a15_afe3_e2ee5f104b85.slice/crio-cfe8b9e226deac52f8dcef4938f1fbe73260ed65ce69968abb479d0ac8abdc10 WatchSource:0}: Error finding container cfe8b9e226deac52f8dcef4938f1fbe73260ed65ce69968abb479d0ac8abdc10: Status 404 returned error can't find the container with id cfe8b9e226deac52f8dcef4938f1fbe73260ed65ce69968abb479d0ac8abdc10 Nov 03 23:55:27.839188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.819534 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg"] Nov 03 23:55:27.855067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.852249 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:27.855067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.852341 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:27.855067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:27.852387 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:28.282564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:28.282533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:28.823775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:28.823722 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" event={"ID":"7240cb0b-5340-4a15-afe3-e2ee5f104b85","Type":"ContainerStarted","Data":"d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404"} Nov 03 23:55:28.823985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:28.823792 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" event={"ID":"7240cb0b-5340-4a15-afe3-e2ee5f104b85","Type":"ContainerStarted","Data":"cfe8b9e226deac52f8dcef4938f1fbe73260ed65ce69968abb479d0ac8abdc10"} Nov 03 23:55:28.856346 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:28.856291 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" podStartSLOduration=1.856277543 podStartE2EDuration="1.856277543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:28.853263957 +0000 UTC m=+1547.675860119" watchObservedRunningTime="2026-11-03 23:55:28.856277543 +0000 UTC m=+1547.678873705" Nov 03 23:55:29.032865 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:29.032814 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:29.429487 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:29.429452 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:29.435416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:29.435388 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:29.830116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:29.830077 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:29.831096 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:29.831074 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:55:30.105693 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:30.105595 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:30.213832 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:30.213797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:31.401013 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:31.400977 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:31.526975 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:31.526940 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:31.637229 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:31.637195 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:32.780422 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:32.780388 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:33.213283 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:33.213135 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:33.332111 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:33.331804 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:34.505638 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:34.505547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:35.551094 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:35.548119 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:35.707245 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:55:35.707196 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:55:36.512019 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.511984 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66"] Nov 03 23:55:36.512714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.512686 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:36.517263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.517237 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.525645 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.524349 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-2341ca-serving-cert\"" Nov 03 23:55:36.530635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.530612 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66"] Nov 03 23:55:36.575122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.575096 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsnjm\" (UniqueName: \"kubernetes.io/projected/5b675c79-e14d-440f-b32b-24dd6bd05427-kube-api-access-vsnjm\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.575440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.575238 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.575440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.575325 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b675c79-e14d-440f-b32b-24dd6bd05427-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.575440 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.575372 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.611313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.611273 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:36.677150 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.677114 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.677343 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.677240 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsnjm\" (UniqueName: \"kubernetes.io/projected/5b675c79-e14d-440f-b32b-24dd6bd05427-kube-api-access-vsnjm\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.677408 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.677361 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.677469 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.677452 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b675c79-e14d-440f-b32b-24dd6bd05427-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.678335 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.678283 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b675c79-e14d-440f-b32b-24dd6bd05427-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.680615 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.680582 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.682529 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.682506 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.689387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.689359 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsnjm\" (UniqueName: \"kubernetes.io/projected/5b675c79-e14d-440f-b32b-24dd6bd05427-kube-api-access-vsnjm\") pod \"maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.711390 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.711361 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:36.842778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.842691 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:36.971954 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:36.971922 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:37.137219 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.137187 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66"] Nov 03 23:55:37.138017 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:55:37.137988 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b675c79_e14d_440f_b32b_24dd6bd05427.slice/crio-5705a19a41f11f50fc6aae5aec410ed32a05a2ecfbc1e1a3318f269f9ba4f4bf WatchSource:0}: Error finding container 5705a19a41f11f50fc6aae5aec410ed32a05a2ecfbc1e1a3318f269f9ba4f4bf: Status 404 returned error can't find the container with id 5705a19a41f11f50fc6aae5aec410ed32a05a2ecfbc1e1a3318f269f9ba4f4bf Nov 03 23:55:37.139978 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.139957 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:55:37.417372 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.417284 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj"] Nov 03 23:55:37.423709 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.423604 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.428908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.428874 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj"] Nov 03 23:55:37.433783 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.433758 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-8093f5-openshift-default-dockercfg-fkkkn\"" Nov 03 23:55:37.465996 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.465961 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:37.500405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.500378 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-envoy\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.500581 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.500423 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.500581 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.500518 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-credential-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.500700 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.500596 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istiod-ca-cert\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.500781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.500759 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-podinfo\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.500879 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.500867 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4r2m\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-kube-api-access-x4r2m\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.500919 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.500904 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-data\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.500954 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.500920 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-certs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.501005 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.500969 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-token\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.602256 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602217 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4r2m\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-kube-api-access-x4r2m\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.602757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602274 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-data\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.602757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602393 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-certs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.602757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602489 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-token\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.602757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602577 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-envoy\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.602757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602627 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.602757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602666 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-data\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.602757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602680 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-credential-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.602757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602697 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-certs\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.603212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602780 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istiod-ca-cert\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.603212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602863 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-podinfo\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.603212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.602969 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.603212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.603071 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-credential-socket\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.603473 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.603453 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istiod-ca-cert\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.604880 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.604862 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-envoy\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.605192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.605175 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-podinfo\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.644119 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.644081 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4r2m\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-kube-api-access-x4r2m\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.646503 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.646476 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-token\") pod \"e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.704162 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.702084 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:37.743122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.742574 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:37.803099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.801903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:37.976405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.976369 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" event={"ID":"5b675c79-e14d-440f-b32b-24dd6bd05427","Type":"ContainerStarted","Data":"665111bb2f9b03e52d6529d54b6eb4401052ae8d0e20f5573c1726dfcec4fa64"} Nov 03 23:55:37.976604 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.976429 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" event={"ID":"5b675c79-e14d-440f-b32b-24dd6bd05427","Type":"ContainerStarted","Data":"5705a19a41f11f50fc6aae5aec410ed32a05a2ecfbc1e1a3318f269f9ba4f4bf"} Nov 03 23:55:37.979228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:37.979141 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:38.000113 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:38.000052 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" podStartSLOduration=1.434264671 podStartE2EDuration="2.000014789s" totalImagesPullingTime="565.750118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:37.998992558 +0000 UTC m=+1556.821588719" watchObservedRunningTime="2026-11-03 23:55:38.000014789 +0000 UTC m=+1556.822610987" Nov 03 23:55:38.087999 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:38.087965 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj"] Nov 03 23:55:38.088693 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:55:38.088659 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8ed697_4cb5_49c9_b618_fbc6f91c7b0f.slice/crio-8a8bc403088ce68d24a4df23c589cd2d7f3cddbb10dda67d3ad5b5dc318d5216 WatchSource:0}: Error finding container 8a8bc403088ce68d24a4df23c589cd2d7f3cddbb10dda67d3ad5b5dc318d5216: Status 404 returned error can't find the container with id 8a8bc403088ce68d24a4df23c589cd2d7f3cddbb10dda67d3ad5b5dc318d5216 Nov 03 23:55:38.091629 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:38.091595 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:38.091717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:38.091659 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:38.091717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:38.091685 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:38.836756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:38.836719 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:38.985620 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:38.985584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:38.986075 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:38.986050 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" event={"ID":"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f","Type":"ContainerStarted","Data":"16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6"} Nov 03 23:55:38.986236 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:38.986114 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" event={"ID":"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f","Type":"ContainerStarted","Data":"8a8bc403088ce68d24a4df23c589cd2d7f3cddbb10dda67d3ad5b5dc318d5216"} Nov 03 23:55:39.013760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:39.013708 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" podStartSLOduration=2.013694879 podStartE2EDuration="2.013694879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:39.011275519 +0000 UTC m=+1557.833871677" watchObservedRunningTime="2026-11-03 23:55:39.013694879 +0000 UTC m=+1557.836291041" Nov 03 23:55:39.687063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:39.687013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:39.743987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:39.743954 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:39.753875 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:39.753846 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:39.888824 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:39.888789 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:39.994790 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:39.994746 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:40.002642 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:40.002615 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:55:40.783714 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:40.783685 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:41.356976 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:41.356947 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:41.474838 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:41.474802 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:41.927840 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:41.927804 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:42.041926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:42.041865 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:42.148055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:42.146882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:42.268537 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:42.268505 6208 scope.go:122] "RemoveContainer" containerID="9f790ae26feaa81e63116101adafea5bbdb6e8e82b55bd27cbd2199721cc05b9" Nov 03 23:55:43.066648 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:43.066614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:43.417248 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:43.417151 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:43.524352 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:43.524322 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:43.992259 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:43.992232 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 03 23:55:44.230985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:44.230902 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:44.336675 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:44.336552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:44.435990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:44.435955 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:45.239877 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:45.239847 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:46.045336 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:46.045296 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:46.933985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:46.933950 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:47.841579 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:47.841549 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:48.631290 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:48.631252 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:49.601523 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:49.601487 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:49.904753 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:49.904673 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7"] Nov 03 23:55:49.905152 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:49.904903 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" containerID="cri-o://ccc9e4a5c8771bb436f86b6ed5adf4b5b32a31be7c783b2258b68e0d64b4d32b" gracePeriod=30 Nov 03 23:55:50.732686 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:50.732653 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:51.466491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:51.466461 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:51.951508 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:51.951409 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:52.100426 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.100387 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:52.346831 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.346797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:52.497166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.497134 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8"] Nov 03 23:55:52.529692 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.529655 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.537738 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.537709 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-2b1f66-openshift-default-dockercfg-wrq2t\"" Nov 03 23:55:52.553445 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.553414 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8"] Nov 03 23:55:52.596938 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.596839 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-certs\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.597213 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.596981 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c01b66-326c-41c4-b612-5551e8af6e85-istiod-ca-cert\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.597213 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.597136 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.597380 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.597227 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wsw\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-kube-api-access-b6wsw\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.597380 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.597361 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-envoy\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.597479 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.597414 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-data\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.597518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.597481 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-credential-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.597518 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.597509 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c01b66-326c-41c4-b612-5551e8af6e85-istio-podinfo\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.597630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.597604 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-istio-token\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.626214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.626168 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:52.703058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.702805 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-certs\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.703058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.702849 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c01b66-326c-41c4-b612-5551e8af6e85-istiod-ca-cert\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.703058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.702896 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.703058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.702992 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wsw\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-kube-api-access-b6wsw\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.703058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.703060 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-envoy\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.703541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.703092 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-data\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.703541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.703162 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-credential-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.703541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.703186 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c01b66-326c-41c4-b612-5551e8af6e85-istio-podinfo\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.703541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.703292 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-istio-token\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.707401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.704361 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-certs\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.707401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.705437 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-data\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.707401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.705698 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-credential-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.707401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.706356 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-socket\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.707401 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.706686 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c01b66-326c-41c4-b612-5551e8af6e85-istiod-ca-cert\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.720691 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.720658 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-envoy\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.725802 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.725221 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wsw\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-kube-api-access-b6wsw\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.727813 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.727784 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-istio-token\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.733442 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.733415 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c01b66-326c-41c4-b612-5551e8af6e85-istio-podinfo\") pod \"e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:52.869317 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:52.869219 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:53.012083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:53.011055 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:53.182672 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:53.182626 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8"] Nov 03 23:55:53.184186 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:55:53.184125 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c01b66_326c_41c4_b612_5551e8af6e85.slice/crio-79580ca9657aa7436366f27baff17e2888e1980b5461ee97af69db765a80f978 WatchSource:0}: Error finding container 79580ca9657aa7436366f27baff17e2888e1980b5461ee97af69db765a80f978: Status 404 returned error can't find the container with id 79580ca9657aa7436366f27baff17e2888e1980b5461ee97af69db765a80f978 Nov 03 23:55:53.187923 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:53.187849 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:53.188078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:53.187948 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:53.188078 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:53.187994 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:55:53.512168 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:53.512131 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:54.143122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.143078 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" event={"ID":"84c01b66-326c-41c4-b612-5551e8af6e85","Type":"ContainerStarted","Data":"62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293"} Nov 03 23:55:54.143304 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.143146 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" event={"ID":"84c01b66-326c-41c4-b612-5551e8af6e85","Type":"ContainerStarted","Data":"79580ca9657aa7436366f27baff17e2888e1980b5461ee97af69db765a80f978"} Nov 03 23:55:54.173156 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.173106 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" podStartSLOduration=2.173091972 podStartE2EDuration="2.173091972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:54.16978028 +0000 UTC m=+1572.992376444" watchObservedRunningTime="2026-11-03 23:55:54.173091972 +0000 UTC m=+1572.995688134" Nov 03 23:55:54.470715 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.470680 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p"] Nov 03 23:55:54.476227 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.476201 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.484134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.484109 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p"] Nov 03 23:55:54.537627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.537591 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa02593-a635-47a7-a21f-9d67b1e8018e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.538144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.537793 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk47l\" (UniqueName: \"kubernetes.io/projected/6fa02593-a635-47a7-a21f-9d67b1e8018e-kube-api-access-lk47l\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.538144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.537846 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.538144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.537928 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.540811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.540784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:54.573390 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.573350 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-8093f5-serving-cert\"" Nov 03 23:55:54.639474 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.639442 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk47l\" (UniqueName: \"kubernetes.io/projected/6fa02593-a635-47a7-a21f-9d67b1e8018e-kube-api-access-lk47l\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.639649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.639497 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.639649 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.639536 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.639721 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.639690 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa02593-a635-47a7-a21f-9d67b1e8018e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.640368 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.640340 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa02593-a635-47a7-a21f-9d67b1e8018e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.642226 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.642202 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.642341 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.642204 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.654527 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.654502 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk47l\" (UniqueName: \"kubernetes.io/projected/6fa02593-a635-47a7-a21f-9d67b1e8018e-kube-api-access-lk47l\") pod \"maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.675142 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.675111 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:54.800500 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.800403 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:54.870265 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.870087 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:54.878236 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.878081 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:54.903093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:54.897986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:55.105354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.105316 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p"] Nov 03 23:55:55.109361 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:55:55.109321 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa02593_a635_47a7_a21f_9d67b1e8018e.slice/crio-bd39b99e09491a15d3e0bf6cb4660095cd570069ba50ecabb535003077e94051 WatchSource:0}: Error finding container bd39b99e09491a15d3e0bf6cb4660095cd570069ba50ecabb535003077e94051: Status 404 returned error can't find the container with id bd39b99e09491a15d3e0bf6cb4660095cd570069ba50ecabb535003077e94051 Nov 03 23:55:55.153235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.153204 6208 generic.go:356] "Generic (PLEG): container finished" podID="22176409-e847-4d2e-bf20-487bf70a69d7" containerID="ccc9e4a5c8771bb436f86b6ed5adf4b5b32a31be7c783b2258b68e0d64b4d32b" exitCode=0 Nov 03 23:55:55.153377 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.153275 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" event={"ID":"22176409-e847-4d2e-bf20-487bf70a69d7","Type":"ContainerDied","Data":"ccc9e4a5c8771bb436f86b6ed5adf4b5b32a31be7c783b2258b68e0d64b4d32b"} Nov 03 23:55:55.154630 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.154604 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" event={"ID":"6fa02593-a635-47a7-a21f-9d67b1e8018e","Type":"ContainerStarted","Data":"bd39b99e09491a15d3e0bf6cb4660095cd570069ba50ecabb535003077e94051"} Nov 03 23:55:55.154891 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.154873 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:55.156192 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.156163 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:55:55.188617 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.177345 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:55:55.258044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.256206 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-credential-socket\") pod \"22176409-e847-4d2e-bf20-487bf70a69d7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " Nov 03 23:55:55.258044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.256289 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22176409-e847-4d2e-bf20-487bf70a69d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22176409-e847-4d2e-bf20-487bf70a69d7-istio-podinfo\") pod \"22176409-e847-4d2e-bf20-487bf70a69d7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " Nov 03 23:55:55.258044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.256397 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22176409-e847-4d2e-bf20-487bf70a69d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22176409-e847-4d2e-bf20-487bf70a69d7-istiod-ca-cert\") pod \"22176409-e847-4d2e-bf20-487bf70a69d7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " Nov 03 23:55:55.258044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.256475 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-envoy\") pod \"22176409-e847-4d2e-bf20-487bf70a69d7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " Nov 03 23:55:55.258044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.256512 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-certs\") pod \"22176409-e847-4d2e-bf20-487bf70a69d7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " Nov 03 23:55:55.258044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.256571 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-socket\") pod \"22176409-e847-4d2e-bf20-487bf70a69d7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " Nov 03 23:55:55.258044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.256743 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-istio-token\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-istio-token\") pod \"22176409-e847-4d2e-bf20-487bf70a69d7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " Nov 03 23:55:55.258044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.256795 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-data\") pod \"22176409-e847-4d2e-bf20-487bf70a69d7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " Nov 03 23:55:55.258044 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.256843 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-kube-api-access-c6s9j\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-kube-api-access-c6s9j\") pod \"22176409-e847-4d2e-bf20-487bf70a69d7\" (UID: \"22176409-e847-4d2e-bf20-487bf70a69d7\") " Nov 03 23:55:55.258665 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.258189 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-credential-socket" pod "22176409-e847-4d2e-bf20-487bf70a69d7" (UID: "22176409-e847-4d2e-bf20-487bf70a69d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:55:55.260322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.259722 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-data" pod "22176409-e847-4d2e-bf20-487bf70a69d7" (UID: "22176409-e847-4d2e-bf20-487bf70a69d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:55:55.260322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.259933 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-socket" pod "22176409-e847-4d2e-bf20-487bf70a69d7" (UID: "22176409-e847-4d2e-bf20-487bf70a69d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:55:55.260322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.260149 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-certs" pod "22176409-e847-4d2e-bf20-487bf70a69d7" (UID: "22176409-e847-4d2e-bf20-487bf70a69d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:55:55.260322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.260294 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22176409-e847-4d2e-bf20-487bf70a69d7-istiod-ca-cert" pod "22176409-e847-4d2e-bf20-487bf70a69d7" (UID: "22176409-e847-4d2e-bf20-487bf70a69d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:55:55.262779 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.262733 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-kube-api-access-c6s9j" pod "22176409-e847-4d2e-bf20-487bf70a69d7" (UID: "22176409-e847-4d2e-bf20-487bf70a69d7"). InnerVolumeSpecName "kube-api-access-c6s9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:55:55.263098 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.263052 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22176409-e847-4d2e-bf20-487bf70a69d7-istio-podinfo" pod "22176409-e847-4d2e-bf20-487bf70a69d7" (UID: "22176409-e847-4d2e-bf20-487bf70a69d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:55:55.263370 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.263348 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-istio-token" pod "22176409-e847-4d2e-bf20-487bf70a69d7" (UID: "22176409-e847-4d2e-bf20-487bf70a69d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:55:55.264717 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.264691 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-envoy" pod "22176409-e847-4d2e-bf20-487bf70a69d7" (UID: "22176409-e847-4d2e-bf20-487bf70a69d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:55:55.363422 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.363334 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:55:55.363722 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.363688 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:55:55.363854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.363727 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6s9j\" (UniqueName: \"kubernetes.io/projected/22176409-e847-4d2e-bf20-487bf70a69d7-kube-api-access-c6s9j\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:55:55.363854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.363746 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:55:55.363854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.363759 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22176409-e847-4d2e-bf20-487bf70a69d7-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:55:55.363854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.363775 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22176409-e847-4d2e-bf20-487bf70a69d7-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:55:55.363854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.363789 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:55:55.363854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.363802 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:55:55.363854 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:55.363816 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22176409-e847-4d2e-bf20-487bf70a69d7-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:55:56.147251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.147201 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:56.185478 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.185442 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" event={"ID":"6fa02593-a635-47a7-a21f-9d67b1e8018e","Type":"ContainerStarted","Data":"ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea"} Nov 03 23:55:56.187241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.187150 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:55:56.190883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.189249 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" Nov 03 23:55:56.190883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.189290 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7" event={"ID":"22176409-e847-4d2e-bf20-487bf70a69d7","Type":"ContainerDied","Data":"164add30b46e84a9c2f33b4a3e0aa600009d356796e5f3673d6f9af166a4424b"} Nov 03 23:55:56.190883 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.189353 6208 scope.go:122] "RemoveContainer" containerID="ccc9e4a5c8771bb436f86b6ed5adf4b5b32a31be7c783b2258b68e0d64b4d32b" Nov 03 23:55:56.277083 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.273809 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" podStartSLOduration=1.8369619130000001 podStartE2EDuration="2.273784758s" totalImagesPullingTime="436.822845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:55:56.272467278 +0000 UTC m=+1575.095063452" watchObservedRunningTime="2026-11-03 23:55:56.273784758 +0000 UTC m=+1575.096380920" Nov 03 23:55:56.329763 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.329730 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7"] Nov 03 23:55:56.338613 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.338525 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-lfks7"] Nov 03 23:55:56.723367 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.723333 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:56.897105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:56.891052 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:57.318945 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:57.318904 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" path="/var/lib/kubelet/pods/22176409-e847-4d2e-bf20-487bf70a69d7/volumes" Nov 03 23:55:57.967063 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:57.966850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:58.346820 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:58.346545 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:58.494985 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:58.494949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:55:59.526046 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:55:59.525997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:00.702145 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:00.702107 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:01.010141 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:01.010013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:02.361385 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:02.361346 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:03.329287 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:03.329258 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 03 23:56:03.725208 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:03.725170 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:04.561405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:04.561251 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:04.763242 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:04.763171 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:56:04.763744 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:04.763262 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:56:05.549027 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:05.548985 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:05.709019 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:56:05.708984 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:56:06.302290 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:06.302253 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:06.411674 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:06.411637 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:06.807868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:06.807834 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:07.311117 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:07.311082 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:07.458443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:07.448418 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:07.897362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:07.897314 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl"] Nov 03 23:56:07.898337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:07.898310 6208 cpu_manager.go:414] "containerMap: removing container" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" Nov 03 23:56:07.898337 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:07.898335 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" Nov 03 23:56:07.898576 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:07.898553 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="22176409-e847-4d2e-bf20-487bf70a69d7" containerName="istio-proxy" Nov 03 23:56:07.983151 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:07.981117 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:08.001184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.000799 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl"] Nov 03 23:56:08.001184 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.000951 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.013685 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.013332 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-15b02a-openshift-default-dockercfg-nsn9f\"" Nov 03 23:56:08.087396 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.087355 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zc7\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-kube-api-access-w6zc7\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.087586 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.087520 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-istio-token\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.087663 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.087586 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfd5078-0143-4c64-8857-e76da9d546de-istiod-ca-cert\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.087723 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.087681 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-certs\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.087785 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.087759 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-envoy\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.087867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.087803 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.087867 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.087831 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-credential-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.088504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.087920 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-data\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.088504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.087994 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfd5078-0143-4c64-8857-e76da9d546de-istio-podinfo\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.137779 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.137744 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:08.190413 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.190319 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-data\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.190413 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.190382 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfd5078-0143-4c64-8857-e76da9d546de-istio-podinfo\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.190718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.190543 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zc7\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-kube-api-access-w6zc7\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.190718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.190630 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-istio-token\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.190718 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.190686 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfd5078-0143-4c64-8857-e76da9d546de-istiod-ca-cert\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.190886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.190716 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-certs\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.190886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.190782 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-envoy\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.190886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.190826 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.190886 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.190852 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-credential-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.191467 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.191440 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-credential-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.191692 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.191672 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-data\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.193134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.192738 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-certs\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.193134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.192942 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfd5078-0143-4c64-8857-e76da9d546de-istiod-ca-cert\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.193134 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.193019 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-socket\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.212978 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.212948 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-envoy\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.213456 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.213419 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfd5078-0143-4c64-8857-e76da9d546de-istio-podinfo\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.215469 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.215449 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-istio-token\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.216238 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.216212 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zc7\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-kube-api-access-w6zc7\") pod \"e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.460504 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.460469 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:08.746919 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.746839 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:08.929921 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.929889 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl"] Nov 03 23:56:08.931181 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:56:08.931150 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfd5078_0143_4c64_8857_e76da9d546de.slice/crio-685c607bf6985ed33ebc5072d8aea97b5ec1d64d5fe9893c6bd46c672f8689da WatchSource:0}: Error finding container 685c607bf6985ed33ebc5072d8aea97b5ec1d64d5fe9893c6bd46c672f8689da: Status 404 returned error can't find the container with id 685c607bf6985ed33ebc5072d8aea97b5ec1d64d5fe9893c6bd46c672f8689da Nov 03 23:56:08.934823 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.934783 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:08.934937 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.934868 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:08.934937 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:08.934918 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:09.487576 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:09.487533 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" event={"ID":"3dfd5078-0143-4c64-8857-e76da9d546de","Type":"ContainerStarted","Data":"c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc"} Nov 03 23:56:09.488152 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:09.487603 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" event={"ID":"3dfd5078-0143-4c64-8857-e76da9d546de","Type":"ContainerStarted","Data":"685c607bf6985ed33ebc5072d8aea97b5ec1d64d5fe9893c6bd46c672f8689da"} Nov 03 23:56:09.551939 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:09.549994 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" podStartSLOduration=2.5499719929999998 podStartE2EDuration="2.549971993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:09.54652262 +0000 UTC m=+1588.369118784" watchObservedRunningTime="2026-11-03 23:56:09.549971993 +0000 UTC m=+1588.372568152" Nov 03 23:56:09.754052 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:09.753685 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:10.463635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:10.461851 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:10.472990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:10.472957 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:10.510681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:10.507125 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:10.510681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:10.510596 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:56:10.790988 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:10.790856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:10.813781 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:10.813748 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:56:10.814053 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:10.814009 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:56:10.947420 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:10.947383 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:11.452419 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:11.452381 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:11.955834 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:11.955794 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:12.099002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.098966 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:12.231099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.231068 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:12.528607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.528572 6208 generic.go:356] "Generic (PLEG): container finished" podID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerID="d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404" exitCode=0 Nov 03 23:56:12.528826 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.528645 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" event={"ID":"7240cb0b-5340-4a15-afe3-e2ee5f104b85","Type":"ContainerDied","Data":"d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404"} Nov 03 23:56:12.529234 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.529205 6208 scope.go:122] "RemoveContainer" containerID="d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404" Nov 03 23:56:12.530708 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.530678 6208 generic.go:356] "Generic (PLEG): container finished" podID="257b95e5-b863-4444-9a35-efd3ca15e157" containerID="6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e" exitCode=0 Nov 03 23:56:12.530851 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.530735 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" event={"ID":"257b95e5-b863-4444-9a35-efd3ca15e157","Type":"ContainerDied","Data":"6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e"} Nov 03 23:56:12.531220 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.531200 6208 scope.go:122] "RemoveContainer" containerID="6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e" Nov 03 23:56:12.531379 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.531278 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:12.531379 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.531366 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:12.531511 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.531419 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:12.537179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.537140 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:12.537295 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.537227 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:12.537295 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.537277 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:12.537863 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.537836 6208 generic.go:356] "Generic (PLEG): container finished" podID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerID="30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081" exitCode=0 Nov 03 23:56:12.537979 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.537955 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" event={"ID":"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0","Type":"ContainerDied","Data":"30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081"} Nov 03 23:56:12.538553 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.538532 6208 scope.go:122] "RemoveContainer" containerID="30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081" Nov 03 23:56:12.541068 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.541009 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:12.541176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.541095 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:12.541176 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:12.541151 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:13.297763 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:13.297705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:13.483579 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:13.479461 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:13.549828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:13.549726 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" event={"ID":"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0","Type":"ContainerStarted","Data":"7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04"} Nov 03 23:56:13.552689 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:13.552663 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" event={"ID":"7240cb0b-5340-4a15-afe3-e2ee5f104b85","Type":"ContainerStarted","Data":"674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b"} Nov 03 23:56:13.564053 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:13.563523 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" event={"ID":"257b95e5-b863-4444-9a35-efd3ca15e157","Type":"ContainerStarted","Data":"7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359"} Nov 03 23:56:13.726054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:13.723708 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:56:13.726054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:13.723831 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 23:56:13.726054 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:13.723888 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 23:56:14.254055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.248212 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:56:14.254055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.249656 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 23:56:14.254055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.249716 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 23:56:14.439064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.433052 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:56:14.439064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.435912 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 23:56:14.439064 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.435979 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 23:56:14.746301 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.746115 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:14.773061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.750747 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 23:56:14.773061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.750803 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 23:56:14.897069 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:14.871150 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:56:15.192117 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:15.191803 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:15.295059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:15.283260 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 23:56:15.295059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:15.283348 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 23:56:15.442077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:15.438027 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 23:56:15.442077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:15.438149 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 23:56:15.728773 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:15.728736 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 23:56:15.731952 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:15.731914 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 23:56:15.907594 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:15.907557 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:16.250286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:16.250240 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 23:56:16.250627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:16.250590 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 23:56:16.439829 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:16.439793 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 23:56:16.440156 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:16.440121 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 23:56:16.730056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:16.723529 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 23:56:16.730056 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:16.723588 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 23:56:16.898067 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:16.897957 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:17.114817 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:17.114722 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:17.249069 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:17.249011 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 23:56:17.249324 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:17.249291 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 23:56:17.437179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:17.437082 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 23:56:17.437179 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:17.437143 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 23:56:17.748338 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:17.741205 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 23:56:17.748338 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:17.741260 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 23:56:17.903694 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:17.903613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:18.253289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:18.252637 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 23:56:18.253289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:18.252705 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 23:56:18.439622 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:18.435399 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 23:56:18.439622 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:18.435462 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 23:56:18.721760 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:18.721714 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:56:18.723798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:18.723771 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 23:56:18.724060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:18.724015 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 23:56:18.985116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:18.982987 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:19.249160 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:19.249058 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Nov 03 23:56:19.249160 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:19.249126 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Nov 03 23:56:19.434077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:19.433555 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 03 23:56:19.434077 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:19.433633 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 03 23:56:19.878107 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:19.878064 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:19.936994 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:19.936959 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:56:19.938776 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:19.938754 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:56:19.940407 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:19.940385 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 03 23:56:20.031710 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:20.031676 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:20.297567 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:20.297519 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:56:20.299403 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:20.299382 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 03 23:56:20.433948 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:20.433918 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:56:20.434797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:20.434774 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 03 23:56:21.128668 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:21.128629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:22.036122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:22.036091 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:22.770767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:22.770737 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:23.640655 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:23.640606 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:24.594055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:24.591489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:25.454379 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:25.454270 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:26.205087 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:26.205052 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:26.883876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:26.883827 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w"] Nov 03 23:56:26.889608 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:26.889580 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:26.895771 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:26.894363 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-15b02a-serving-cert\"" Nov 03 23:56:26.903682 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:26.903646 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w"] Nov 03 23:56:26.986918 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:26.986888 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:27.018060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.017999 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/e6a19ace-99af-4d67-b658-12655bc033cb-kube-api-access-r5qn7\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.018249 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.018132 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.018249 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.018168 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.018341 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.018319 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a19ace-99af-4d67-b658-12655bc033cb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.123251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.123211 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/e6a19ace-99af-4d67-b658-12655bc033cb-kube-api-access-r5qn7\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.123971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.123941 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.128198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.126751 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.128198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.126887 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a19ace-99af-4d67-b658-12655bc033cb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.128198 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.127815 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a19ace-99af-4d67-b658-12655bc033cb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.128571 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.128549 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.135647 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.134538 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.139118 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.139087 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/e6a19ace-99af-4d67-b658-12655bc033cb-kube-api-access-r5qn7\") pod \"maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.148792 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.148753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:27.217173 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.217131 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:27.670381 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.667556 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w"] Nov 03 23:56:27.921262 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:27.921166 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" event={"ID":"e6a19ace-99af-4d67-b658-12655bc033cb","Type":"ContainerStarted","Data":"a5531c865ea33bdcb1fea40856614ee0b707cc5764505fcc0bac60c9855dbd30"} Nov 03 23:56:28.015701 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:28.015669 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:28.936859 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:28.936822 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:28.944735 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:28.944685 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" event={"ID":"e6a19ace-99af-4d67-b658-12655bc033cb","Type":"ContainerStarted","Data":"bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d"} Nov 03 23:56:28.944914 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:28.944882 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:28.985315 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:28.985251 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" podStartSLOduration=2.528322122 podStartE2EDuration="2.985230888s" totalImagesPullingTime="456.908766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:56:28.967468207 +0000 UTC m=+1607.790064368" watchObservedRunningTime="2026-11-03 23:56:28.985230888 +0000 UTC m=+1607.807827051" Nov 03 23:56:29.105815 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:29.105784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:30.029240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:30.029200 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:30.950896 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:30.950849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:31.834711 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:31.834662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:32.214310 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:32.212425 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:32.664566 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:32.664477 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:33.723525 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:33.723487 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:34.006605 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:34.006523 6208 generic.go:356] "Generic (PLEG): container finished" podID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerID="16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6" exitCode=0 Nov 03 23:56:34.006804 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:34.006602 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" event={"ID":"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f","Type":"ContainerDied","Data":"16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6"} Nov 03 23:56:34.007117 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:34.007103 6208 scope.go:122] "RemoveContainer" containerID="16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6" Nov 03 23:56:34.008891 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:34.008857 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:34.008977 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:34.008937 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:34.008977 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:34.008970 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:56:34.573060 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:34.553817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:34.985798 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:34.985768 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 03 23:56:35.002862 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:35.002822 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:35.016853 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:35.016821 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" event={"ID":"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f","Type":"ContainerStarted","Data":"7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746"} Nov 03 23:56:35.549596 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:35.549562 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:35.739632 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:56:35.739593 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:56:35.743889 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:35.743869 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:56:35.745959 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:35.745934 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 23:56:35.746114 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:35.745980 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 23:56:36.416427 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:36.416360 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:36.743806 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:36.743761 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 23:56:36.743987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:36.743830 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 23:56:37.149338 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:37.149246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:37.743253 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:37.743219 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 23:56:37.743741 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:37.743271 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 23:56:37.833369 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:37.833336 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:38.050190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:38.050096 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:38.743241 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:38.743208 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 23:56:38.743422 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:38.743263 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 23:56:38.842137 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:38.842097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:39.417708 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:39.417678 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt"] Nov 03 23:56:39.417890 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:39.417872 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" podUID="19b457bf-87e3-418f-bed9-8defb674e773" containerName="istio-proxy" containerID="cri-o://7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450" gracePeriod=30 Nov 03 23:56:39.726354 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:39.726313 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:39.743635 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:39.743598 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 23:56:39.744143 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:39.743655 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 23:56:39.995190 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:39.995084 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:56:40.528378 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:40.528348 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:40.748967 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:40.748934 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:56:40.750146 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:40.750125 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 03 23:56:41.349377 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:41.349336 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:42.332414 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:42.332379 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:43.124043 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:43.124005 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:43.928571 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:43.928532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:44.041881 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.041841 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:44.707880 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.707852 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:56:44.810468 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.810428 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-credential-socket\") pod \"19b457bf-87e3-418f-bed9-8defb674e773\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " Nov 03 23:56:44.810653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.810544 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19b457bf-87e3-418f-bed9-8defb674e773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b457bf-87e3-418f-bed9-8defb674e773-istio-podinfo\") pod \"19b457bf-87e3-418f-bed9-8defb674e773\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " Nov 03 23:56:44.810653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.810609 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-istio-token\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-istio-token\") pod \"19b457bf-87e3-418f-bed9-8defb674e773\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " Nov 03 23:56:44.810653 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.810651 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-data\") pod \"19b457bf-87e3-418f-bed9-8defb674e773\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " Nov 03 23:56:44.810796 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.810722 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-credential-socket" pod "19b457bf-87e3-418f-bed9-8defb674e773" (UID: "19b457bf-87e3-418f-bed9-8defb674e773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:56:44.810796 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.810738 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-socket\") pod \"19b457bf-87e3-418f-bed9-8defb674e773\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " Nov 03 23:56:44.810903 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.810837 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-envoy\") pod \"19b457bf-87e3-418f-bed9-8defb674e773\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " Nov 03 23:56:44.810964 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.810948 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-certs\") pod \"19b457bf-87e3-418f-bed9-8defb674e773\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " Nov 03 23:56:44.811075 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811021 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-data" pod "19b457bf-87e3-418f-bed9-8defb674e773" (UID: "19b457bf-87e3-418f-bed9-8defb674e773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:56:44.811167 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.810965 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-socket" pod "19b457bf-87e3-418f-bed9-8defb674e773" (UID: "19b457bf-87e3-418f-bed9-8defb674e773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:56:44.811167 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811077 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b457bf-87e3-418f-bed9-8defb674e773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b457bf-87e3-418f-bed9-8defb674e773-istiod-ca-cert\") pod \"19b457bf-87e3-418f-bed9-8defb674e773\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " Nov 03 23:56:44.811281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811191 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-kube-api-access-p2599\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-kube-api-access-p2599\") pod \"19b457bf-87e3-418f-bed9-8defb674e773\" (UID: \"19b457bf-87e3-418f-bed9-8defb674e773\") " Nov 03 23:56:44.811336 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811300 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-certs" pod "19b457bf-87e3-418f-bed9-8defb674e773" (UID: "19b457bf-87e3-418f-bed9-8defb674e773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:56:44.811390 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811371 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b457bf-87e3-418f-bed9-8defb674e773-istiod-ca-cert" pod "19b457bf-87e3-418f-bed9-8defb674e773" (UID: "19b457bf-87e3-418f-bed9-8defb674e773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:56:44.811928 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811893 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b457bf-87e3-418f-bed9-8defb674e773-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:56:44.811928 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811922 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:56:44.812133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811939 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:56:44.812133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811954 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:56:44.812133 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.811967 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:56:44.812969 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.812942 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19b457bf-87e3-418f-bed9-8defb674e773-istio-podinfo" pod "19b457bf-87e3-418f-bed9-8defb674e773" (UID: "19b457bf-87e3-418f-bed9-8defb674e773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:56:44.813304 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.813276 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-istio-token" pod "19b457bf-87e3-418f-bed9-8defb674e773" (UID: "19b457bf-87e3-418f-bed9-8defb674e773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:56:44.813409 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.813334 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-kube-api-access-p2599" pod "19b457bf-87e3-418f-bed9-8defb674e773" (UID: "19b457bf-87e3-418f-bed9-8defb674e773"). InnerVolumeSpecName "kube-api-access-p2599". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:56:44.813409 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.813387 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-envoy" pod "19b457bf-87e3-418f-bed9-8defb674e773" (UID: "19b457bf-87e3-418f-bed9-8defb674e773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:56:44.843752 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.843711 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:44.914382 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.914344 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b457bf-87e3-418f-bed9-8defb674e773-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:56:44.914382 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.914379 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:56:44.914382 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.914388 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b457bf-87e3-418f-bed9-8defb674e773-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:56:44.914780 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:44.914397 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2599\" (UniqueName: \"kubernetes.io/projected/19b457bf-87e3-418f-bed9-8defb674e773-kube-api-access-p2599\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 23:56:45.133002 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.132962 6208 generic.go:356] "Generic (PLEG): container finished" podID="19b457bf-87e3-418f-bed9-8defb674e773" containerID="7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450" exitCode=0 Nov 03 23:56:45.133485 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.133026 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" Nov 03 23:56:45.133485 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.133054 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" event={"ID":"19b457bf-87e3-418f-bed9-8defb674e773","Type":"ContainerDied","Data":"7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450"} Nov 03 23:56:45.133485 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.133131 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt" event={"ID":"19b457bf-87e3-418f-bed9-8defb674e773","Type":"ContainerDied","Data":"f943e2e6d6e93a1af9bb2e6feada274fbeea711b576135d7398827033b1c2ed5"} Nov 03 23:56:45.133485 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.133155 6208 scope.go:122] "RemoveContainer" containerID="7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450" Nov 03 23:56:45.144507 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.144484 6208 scope.go:122] "RemoveContainer" containerID="7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450" Nov 03 23:56:45.144825 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:56:45.144807 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450\": container with ID starting with 7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450 not found: ID does not exist" containerID="7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450" Nov 03 23:56:45.144874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.144836 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450"} err="failed to get container status \"7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450\": rpc error: code = NotFound desc = could not find container \"7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450\": container with ID starting with 7ec6e947b1e7f07f5afb3f82a79a20eb332b38811555333a481214764ac21450 not found: ID does not exist" Nov 03 23:56:45.158424 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.158389 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt"] Nov 03 23:56:45.165990 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.165960 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-b86dt"] Nov 03 23:56:45.298281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.298237 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b457bf-87e3-418f-bed9-8defb674e773" path="/var/lib/kubelet/pods/19b457bf-87e3-418f-bed9-8defb674e773/volumes" Nov 03 23:56:45.658794 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:45.658761 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:46.533812 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:46.533780 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:47.428550 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:47.428509 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:47.576380 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:47.576333 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:48.243731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:48.243701 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:49.141767 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:49.141733 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:50.258527 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:50.258332 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:51.340521 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:51.339936 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:51.989166 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:51.989133 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:52.109121 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:52.109084 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:53.098061 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:53.097982 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:53.938432 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:53.938392 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:54.738809 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:54.738774 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:55.650053 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:55.649936 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:56.437471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:56.437428 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:57.326286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:57.326244 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:57.426876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:57.426761 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:58.185542 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:58.185507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:58.938043 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:58.937991 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:56:59.745144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:56:59.745096 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:00.536462 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:00.536423 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:01.331004 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.330966 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:01.403511 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.403480 6208 generic.go:356] "Generic (PLEG): container finished" podID="84c01b66-326c-41c4-b612-5551e8af6e85" containerID="62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293" exitCode=0 Nov 03 23:57:01.403688 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.403552 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" event={"ID":"84c01b66-326c-41c4-b612-5551e8af6e85","Type":"ContainerDied","Data":"62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293"} Nov 03 23:57:01.404105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.404082 6208 scope.go:122] "RemoveContainer" containerID="62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293" Nov 03 23:57:01.405676 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.405650 6208 generic.go:356] "Generic (PLEG): container finished" podID="3dfd5078-0143-4c64-8857-e76da9d546de" containerID="c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc" exitCode=0 Nov 03 23:57:01.405790 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.405726 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" event={"ID":"3dfd5078-0143-4c64-8857-e76da9d546de","Type":"ContainerDied","Data":"c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc"} Nov 03 23:57:01.406132 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.406079 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:57:01.406251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.406163 6208 scope.go:122] "RemoveContainer" containerID="c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc" Nov 03 23:57:01.406251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.406171 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:57:01.406251 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.406206 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:57:01.407950 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.407900 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:57:01.408081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.408048 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:57:01.408135 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.408122 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:57:01.763874 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:01.762631 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:02.349406 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:02.349368 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:02.509058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:02.463884 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" event={"ID":"3dfd5078-0143-4c64-8857-e76da9d546de","Type":"ContainerStarted","Data":"98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34"} Nov 03 23:57:02.509058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:02.466456 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:57:02.509058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:02.466556 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 23:57:02.509058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:02.466599 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 23:57:02.509058 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:02.471372 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" event={"ID":"84c01b66-326c-41c4-b612-5551e8af6e85","Type":"ContainerStarted","Data":"b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472"} Nov 03 23:57:02.889066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:02.887581 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:57:02.889066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:02.887729 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:57:02.889066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:02.887774 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:57:03.450439 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:03.450408 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:03.461449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:03.461413 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 23:57:03.461793 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:03.461751 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 23:57:03.876730 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:03.876680 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:57:03.881097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:03.877004 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:57:04.278057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:04.277279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:04.467057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:04.462763 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 23:57:04.467057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:04.462829 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 23:57:04.875461 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:04.875416 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:57:04.875775 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:04.875744 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:57:05.172856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:05.172742 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:05.323662 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:05.323522 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:05.477057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:05.463435 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 23:57:05.477057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:05.463502 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 23:57:05.817948 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:57:05.817857 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:57:05.887490 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:05.887453 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:57:05.887777 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:05.887749 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:57:06.069111 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:06.067604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:06.183384 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:06.183339 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:06.384025 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:06.383895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:06.463066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:06.461464 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 23:57:06.463066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:06.461525 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 23:57:06.873057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:06.872397 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:57:06.873057 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:06.872464 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:57:07.143848 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:07.143761 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:07.475627 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:07.475585 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 23:57:07.475923 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:07.475890 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 23:57:07.870637 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:07.870526 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:57:07.870637 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:07.870598 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:57:07.939564 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:07.939525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:08.466675 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:08.466638 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:57:08.467066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:08.467010 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:57:08.467740 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:08.467721 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 03 23:57:08.874892 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:08.874814 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:57:08.875313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:08.875282 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:57:08.875846 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:08.875824 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 03 23:57:08.927593 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:08.927562 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:09.770334 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:09.770299 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:10.537545 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:10.537504 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:11.466411 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:11.466377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:11.572868 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:11.572825 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:12.365043 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:12.364979 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:13.151927 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:13.151882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:13.983913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:13.983881 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:14.731893 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:14.731856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:15.599016 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:15.598982 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:16.344007 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:16.343972 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:17.139299 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:17.139262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:18.042458 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:18.042419 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:18.389422 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:18.389347 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:19.038533 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:19.038500 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:19.927996 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:19.927955 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:20.746441 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:20.746410 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:22.213557 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.213519 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:22.451164 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.451096 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs"] Nov 03 23:57:22.496671 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.494334 6208 cpu_manager.go:414] "containerMap: removing container" podUID="19b457bf-87e3-418f-bed9-8defb674e773" containerName="istio-proxy" Nov 03 23:57:22.496671 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.494367 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b457bf-87e3-418f-bed9-8defb674e773" containerName="istio-proxy" Nov 03 23:57:22.496671 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.494588 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b457bf-87e3-418f-bed9-8defb674e773" containerName="istio-proxy" Nov 03 23:57:22.502658 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.501022 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.505965 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.505860 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-58b75d-openshift-default-dockercfg-ks9wt\"" Nov 03 23:57:22.563939 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.551548 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs"] Nov 03 23:57:22.622282 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.584433 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.622677 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.622648 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.631139 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.631106 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.654059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.645253 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgnr\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-kube-api-access-9xgnr\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.654059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.645328 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.654059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.645365 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.654059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.645395 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.654059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.645635 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.654059 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.645700 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.751436 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.751336 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.751697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.751677 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.751946 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.751926 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.752191 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.752172 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.752324 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.752309 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.752443 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.752424 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgnr\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-kube-api-access-9xgnr\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.752586 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.752570 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.752729 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.752707 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.752859 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.752842 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.753730 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.753707 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-credential-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.755446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.755423 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istiod-ca-cert\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.756299 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.755812 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-data\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.756575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.756555 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-certs\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.757200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.757151 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-socket\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.783927 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.778687 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-envoy\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.783927 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.780270 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-podinfo\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.784217 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.783949 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgnr\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-kube-api-access-9xgnr\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.786605 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.786559 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-token\") pod \"e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:22.852111 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:22.852073 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:23.156296 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:23.156203 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:23.163834 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:23.163760 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs"] Nov 03 23:57:23.173953 ip-10-0-138-137 kubenswrapper[6208]: W1103 23:57:23.173912 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29233c7_5e58_4ed7_b12c_6aa0ffe49e79.slice/crio-600d8209a9022f520bef6ed2edb8c6ef5432acb9d1327edad5270e2493a6614a WatchSource:0}: Error finding container 600d8209a9022f520bef6ed2edb8c6ef5432acb9d1327edad5270e2493a6614a: Status 404 returned error can't find the container with id 600d8209a9022f520bef6ed2edb8c6ef5432acb9d1327edad5270e2493a6614a Nov 03 23:57:23.177907 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:23.177864 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:57:23.178070 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:23.177967 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:57:23.178070 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:23.178018 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 23:57:23.906122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:23.902477 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" event={"ID":"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79","Type":"ContainerStarted","Data":"93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc"} Nov 03 23:57:23.906122 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:23.902551 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" event={"ID":"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79","Type":"ContainerStarted","Data":"600d8209a9022f520bef6ed2edb8c6ef5432acb9d1327edad5270e2493a6614a"} Nov 03 23:57:23.981314 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:23.981233 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" podStartSLOduration=1.981185126 podStartE2EDuration="1.981185126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:57:23.977564717 +0000 UTC m=+1662.800160882" watchObservedRunningTime="2026-11-03 23:57:23.981185126 +0000 UTC m=+1662.803781289" Nov 03 23:57:24.313093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:24.312841 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:24.852362 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:24.852330 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:24.878926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:24.878210 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 23:57:24.878926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:24.878320 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 23:57:25.125603 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:25.125452 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:25.852903 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:25.852867 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 23:57:25.853111 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:25.852933 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 23:57:25.879580 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:25.879537 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:26.829046 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:26.828998 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:26.870363 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:26.870332 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 23:57:26.870625 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:26.870596 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 23:57:27.244907 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:27.244870 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:27.778758 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:27.778714 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:27.864079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:27.859973 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 23:57:27.864079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:27.860114 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 23:57:28.646610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:28.646576 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:28.880263 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:28.880210 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 23:57:28.880897 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:28.880838 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 23:57:29.615595 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:29.615556 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:29.892697 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:29.892596 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:29.893652 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:29.893625 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:29.894727 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:29.894701 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 03 23:57:30.350543 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:30.350493 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:31.080413 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:31.080381 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:31.823214 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:31.823180 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:32.549071 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:32.549041 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:33.418120 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:33.418061 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:34.064228 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:34.064186 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:34.963107 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:34.963069 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:35.707768 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:57:35.707728 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:57:35.838003 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:35.837967 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:36.548062 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:36.548011 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:37.340321 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:37.340284 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:38.145813 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:38.145777 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:38.920534 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:38.920500 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:39.751344 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:39.751306 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:40.630125 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:40.630091 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:41.433541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:41.433500 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:42.369330 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:42.369297 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:43.130156 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:43.130118 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:43.968779 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:43.968743 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:44.851934 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:44.851899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:45.639464 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:45.639432 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:46.427313 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:46.427271 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:47.184018 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:47.183985 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:47.957447 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:47.957402 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:48.744200 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:48.744165 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:49.631157 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:49.631122 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:50.352481 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:50.352450 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:51.251188 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:51.251131 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:52.158515 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:52.158478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:53.338506 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:53.338463 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:54.065575 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:54.065532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:54.963856 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:54.963815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:55.834558 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:55.834521 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:56.595097 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:56.595057 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:57.520731 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:57.520697 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:58.748897 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:58.748860 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:58.961607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:58.961571 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:57:59.734986 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:57:59.734951 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:00.529292 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:00.529256 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:01.231681 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:01.231648 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:01.951491 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:01.951458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:02.755212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:02.755175 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:03.745286 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:03.745238 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:04.532839 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:04.532807 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:05.294055 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:05.294006 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:05.706269 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:58:05.706236 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:58:05.956626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:05.956532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:06.423726 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:06.423632 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:06.745662 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:06.745628 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:07.468747 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:07.468702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:08.183944 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:08.183909 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:08.948607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:08.948551 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:09.736774 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:09.736733 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:10.465103 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:10.465067 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:11.356048 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:11.355992 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:11.529240 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:11.529199 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:12.138341 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:12.138301 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:12.918636 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:12.918594 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:13.730517 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:13.730481 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:14.441509 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:14.441473 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:15.305593 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:15.305557 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:16.032556 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:16.032526 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:16.858827 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:16.858787 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:17.644759 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:17.644724 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:18.422144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:18.422107 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:19.325590 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:19.325554 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:20.042677 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:20.042577 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:20.763099 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:20.763060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:21.482100 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:21.482057 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:22.297590 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:22.297556 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:23.144405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:23.144370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:23.864893 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:23.864854 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:24.739670 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:24.739640 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:25.548079 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:25.548021 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:26.331580 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:26.331543 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:27.027289 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:27.027250 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:27.841716 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:27.841680 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:28.757810 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:28.757774 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:29.721598 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:29.721565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:30.551861 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:30.551820 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:31.293910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:31.293874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:32.031339 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:32.031303 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:32.848828 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:32.848792 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:33.628844 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:33.628807 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:34.342757 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:34.342717 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:35.057216 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:35.057172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:35.713982 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:58:35.713942 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:58:35.776285 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:35.776253 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:36.625268 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:36.625178 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:37.150193 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:37.150154 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:37.364548 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:37.364509 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:38.089011 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:38.088974 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:38.944651 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:38.944617 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:39.742460 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:39.742391 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:40.454387 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:40.454347 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:41.220737 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:41.220701 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:41.954066 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:41.954010 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:42.669657 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:42.669548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:43.617544 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:43.617507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:44.627230 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:44.627192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:45.332652 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:45.332612 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:46.044249 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:46.044210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:46.852640 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:46.852584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:47.632876 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:47.632840 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:48.465300 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:48.465265 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:49.165225 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:49.165187 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:50.023126 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:50.023091 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:50.921375 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:50.921338 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:51.739626 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:51.739592 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:52.534667 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:52.534622 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:53.244584 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:53.244544 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:54.042291 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:54.042244 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:54.749949 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:54.749912 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:55.586477 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:55.586435 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:56.419605 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:56.419565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:57.144844 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:57.144809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:58.025741 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:58.025705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:59.038784 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:59.038749 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:58:59.964093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:58:59.963855 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:00.661821 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:00.661783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:01.530167 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:01.530138 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:02.431878 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:02.431841 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:02.706400 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:02.706368 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:03.175416 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:03.175316 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:04.062755 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:04.062716 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:04.841910 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:04.841875 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:05.494778 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:05.494742 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:05.595449 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:05.595405 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:05.708144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:05.707999 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:05.756616 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:59:05.756531 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:59:06.380470 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:06.380437 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:06.507797 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:06.507750 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:07.229971 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:07.229941 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:08.140740 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:08.140709 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:08.873988 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:08.873954 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:09.577621 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:09.577588 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:10.291476 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:10.291441 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:11.142942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:11.142904 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:11.594610 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:11.594568 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:11.991471 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:11.991441 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:12.729405 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:12.729368 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:13.638187 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:13.637980 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:14.342639 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:14.342597 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:15.121942 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:15.121911 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:15.841662 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:15.841633 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:16.598750 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:16.598702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:17.472235 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:17.472201 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:18.393768 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:18.393727 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:19.220322 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:19.220285 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:20.018628 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:20.018588 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:20.807199 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:20.807163 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:21.611589 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:21.611548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:22.612811 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:22.612774 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:23.423913 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:23.423872 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:23.858766 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:23.858731 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:24.398588 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:24.398549 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:25.091710 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:25.091668 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:25.863446 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:25.863414 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:26.647411 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:26.647371 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:27.523329 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:27.523287 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:28.320756 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:28.320721 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:29.088652 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:29.088615 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:29.192420 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:29.192372 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:29.969633 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:29.969592 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:30.834281 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:30.834241 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:31.750900 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:31.750858 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:32.608105 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:32.608065 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:33.432582 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:33.432546 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:34.220632 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:34.220561 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:35.041656 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:35.041618 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:35.705626 ip-10-0-138-137 kubenswrapper[6208]: E1103 23:59:35.705591 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 03 23:59:35.866812 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:35.866777 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:36.737692 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:36.737655 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:37.209541 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:37.209505 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:37.629450 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:37.629353 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:38.428089 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:38.428022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:39.355505 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:39.355468 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:40.059926 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:40.059890 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:40.954116 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:40.954077 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:41.753374 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:41.753339 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:42.234640 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:42.234612 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:59:42.283799 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:42.283768 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 03 23:59:42.473724 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:42.473682 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:43.222607 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:43.222564 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:43.987823 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:43.987783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:44.826469 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:44.826432 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:45.546741 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:45.546710 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:46.286831 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:46.286795 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:47.078827 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:47.078791 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:48.037924 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:48.037889 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:49.134212 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:49.134166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:49.956906 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:49.956806 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:50.686175 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:50.686126 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:50.785987 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:50.785945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:50.901076 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:50.901014 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:51.717534 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:51.717506 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:52.448750 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:52.448718 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:53.240459 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:53.240425 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:54.225908 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:54.225875 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:54.951081 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:54.950995 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:55.869786 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:55.869739 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:56.658211 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:56.658172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:57.369411 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:57.369377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:58.221093 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:58.221028 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:59.026144 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:59.026103 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 03 23:59:59.827305 ip-10-0-138-137 kubenswrapper[6208]: I1103 23:59:59.827273 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:00.196818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.196719 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2"] Nov 04 00:00:00.204005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.203973 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:00.205149 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.205116 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh"] Nov 04 00:00:00.210345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.210317 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:00.225195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.225159 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2"] Nov 04 00:00:00.230603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.230569 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh"] Nov 04 00:00:00.256835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.256777 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2"] Nov 04 00:00:00.265359 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.265310 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:00.268305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.268276 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\"" Nov 04 00:00:00.268541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.268523 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr"] Nov 04 00:00:00.274263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.274238 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:00.281192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.281161 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f57d3c-6e1a-4cb2-82d6-969494d04963-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2\" (UID: \"50f57d3c-6e1a-4cb2-82d6-969494d04963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:00.281336 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.281215 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjlkn\" (UniqueName: \"kubernetes.io/projected/95df9311-7c2c-48b1-b9df-ef7632b87034-kube-api-access-zjlkn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh\" (UID: \"95df9311-7c2c-48b1-b9df-ef7632b87034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:00.281485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.281467 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df9311-7c2c-48b1-b9df-ef7632b87034-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh\" (UID: \"95df9311-7c2c-48b1-b9df-ef7632b87034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:00.281661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.281641 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbhzj\" (UniqueName: \"kubernetes.io/projected/50f57d3c-6e1a-4cb2-82d6-969494d04963-kube-api-access-cbhzj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2\" (UID: \"50f57d3c-6e1a-4cb2-82d6-969494d04963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:00.301903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.301874 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2"] Nov 04 00:00:00.315685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.315658 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk"] Nov 04 00:00:00.320049 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.320014 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:00.329905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.329870 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr"] Nov 04 00:00:00.336129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.336102 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk"] Nov 04 00:00:00.383378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.383327 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a430fb1-1948-4357-87e9-58af7f08b30b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895840-rdbs2\" (UID: \"5a430fb1-1948-4357-87e9-58af7f08b30b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:00.383569 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.383458 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5537b60-f90f-47fd-aac1-433ec053333a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr\" (UID: \"a5537b60-f90f-47fd-aac1-433ec053333a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:00.383569 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.383527 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df9311-7c2c-48b1-b9df-ef7632b87034-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh\" (UID: \"95df9311-7c2c-48b1-b9df-ef7632b87034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:00.383772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.383748 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk\" (UID: \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:00.383841 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.383800 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpk5k\" (UniqueName: \"kubernetes.io/projected/5a430fb1-1948-4357-87e9-58af7f08b30b-kube-api-access-jpk5k\") pod \"maas-api-key-cleanup-29895840-rdbs2\" (UID: \"5a430fb1-1948-4357-87e9-58af7f08b30b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:00.383918 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.383902 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbhzj\" (UniqueName: \"kubernetes.io/projected/50f57d3c-6e1a-4cb2-82d6-969494d04963-kube-api-access-cbhzj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2\" (UID: \"50f57d3c-6e1a-4cb2-82d6-969494d04963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:00.383992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.383971 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c5mk\" (UniqueName: \"kubernetes.io/projected/a5537b60-f90f-47fd-aac1-433ec053333a-kube-api-access-7c5mk\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr\" (UID: \"a5537b60-f90f-47fd-aac1-433ec053333a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:00.384106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.384088 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhff\" (UniqueName: \"kubernetes.io/projected/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-kube-api-access-wmhff\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk\" (UID: \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:00.384176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.384154 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f57d3c-6e1a-4cb2-82d6-969494d04963-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2\" (UID: \"50f57d3c-6e1a-4cb2-82d6-969494d04963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:00.384239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.384219 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjlkn\" (UniqueName: \"kubernetes.io/projected/95df9311-7c2c-48b1-b9df-ef7632b87034-kube-api-access-zjlkn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh\" (UID: \"95df9311-7c2c-48b1-b9df-ef7632b87034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:00.384441 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.384418 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df9311-7c2c-48b1-b9df-ef7632b87034-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh\" (UID: \"95df9311-7c2c-48b1-b9df-ef7632b87034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:00.384809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.384787 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f57d3c-6e1a-4cb2-82d6-969494d04963-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2\" (UID: \"50f57d3c-6e1a-4cb2-82d6-969494d04963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:00.399514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.399481 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbhzj\" (UniqueName: \"kubernetes.io/projected/50f57d3c-6e1a-4cb2-82d6-969494d04963-kube-api-access-cbhzj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2\" (UID: \"50f57d3c-6e1a-4cb2-82d6-969494d04963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:00.399661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.399482 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjlkn\" (UniqueName: \"kubernetes.io/projected/95df9311-7c2c-48b1-b9df-ef7632b87034-kube-api-access-zjlkn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh\" (UID: \"95df9311-7c2c-48b1-b9df-ef7632b87034\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:00.485446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.485409 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk\" (UID: \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:00.485628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.485480 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpk5k\" (UniqueName: \"kubernetes.io/projected/5a430fb1-1948-4357-87e9-58af7f08b30b-kube-api-access-jpk5k\") pod \"maas-api-key-cleanup-29895840-rdbs2\" (UID: \"5a430fb1-1948-4357-87e9-58af7f08b30b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:00.485628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.485575 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5mk\" (UniqueName: \"kubernetes.io/projected/a5537b60-f90f-47fd-aac1-433ec053333a-kube-api-access-7c5mk\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr\" (UID: \"a5537b60-f90f-47fd-aac1-433ec053333a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:00.485628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.485625 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhff\" (UniqueName: \"kubernetes.io/projected/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-kube-api-access-wmhff\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk\" (UID: \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:00.485829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.485747 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a430fb1-1948-4357-87e9-58af7f08b30b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895840-rdbs2\" (UID: \"5a430fb1-1948-4357-87e9-58af7f08b30b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:00.485829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.485812 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5537b60-f90f-47fd-aac1-433ec053333a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr\" (UID: \"a5537b60-f90f-47fd-aac1-433ec053333a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:00.486318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.486291 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk\" (UID: \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:00.486546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.486523 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5537b60-f90f-47fd-aac1-433ec053333a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr\" (UID: \"a5537b60-f90f-47fd-aac1-433ec053333a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:00.486922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.486888 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a430fb1-1948-4357-87e9-58af7f08b30b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895840-rdbs2\" (UID: \"5a430fb1-1948-4357-87e9-58af7f08b30b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:00.500717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.500678 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5mk\" (UniqueName: \"kubernetes.io/projected/a5537b60-f90f-47fd-aac1-433ec053333a-kube-api-access-7c5mk\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr\" (UID: \"a5537b60-f90f-47fd-aac1-433ec053333a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:00.503974 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.503942 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpk5k\" (UniqueName: \"kubernetes.io/projected/5a430fb1-1948-4357-87e9-58af7f08b30b-kube-api-access-jpk5k\") pod \"maas-api-key-cleanup-29895840-rdbs2\" (UID: \"5a430fb1-1948-4357-87e9-58af7f08b30b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:00.517789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.517747 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhff\" (UniqueName: \"kubernetes.io/projected/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-kube-api-access-wmhff\") pod \"maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk\" (UID: \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:00.568311 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.568260 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:00.591697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.591665 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:00.607154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.607120 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:00.619410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.619377 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:00.640443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.640181 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:00.769913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.768023 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:00.919939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:00.919660 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2"] Nov 04 00:00:01.108750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.108715 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk"] Nov 04 00:00:01.113544 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.113515 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh"] Nov 04 00:00:01.149549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.149524 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2"] Nov 04 00:00:01.156474 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:00:01.156440 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26dbe69_425e_4f9e_8758_4d86d7d2aeb3.slice/crio-1c41379c776dbc323da11690948d5f574224fe1ea868e19425fed24a928281d2 WatchSource:0}: Error finding container 1c41379c776dbc323da11690948d5f574224fe1ea868e19425fed24a928281d2: Status 404 returned error can't find the container with id 1c41379c776dbc323da11690948d5f574224fe1ea868e19425fed24a928281d2 Nov 04 00:00:01.157345 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:00:01.157315 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95df9311_7c2c_48b1_b9df_ef7632b87034.slice/crio-efd90b531b3c6e573430f5fa2c96e3f27ad2269fba4ca57ae2f8f96cf3825983 WatchSource:0}: Error finding container efd90b531b3c6e573430f5fa2c96e3f27ad2269fba4ca57ae2f8f96cf3825983: Status 404 returned error can't find the container with id efd90b531b3c6e573430f5fa2c96e3f27ad2269fba4ca57ae2f8f96cf3825983 Nov 04 00:00:01.158512 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:00:01.158489 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a430fb1_1948_4357_87e9_58af7f08b30b.slice/crio-43e603ff1d77f024298721f7335d21856aba6519157251fda24e1e86fc64eb45 WatchSource:0}: Error finding container 43e603ff1d77f024298721f7335d21856aba6519157251fda24e1e86fc64eb45: Status 404 returned error can't find the container with id 43e603ff1d77f024298721f7335d21856aba6519157251fda24e1e86fc64eb45 Nov 04 00:00:01.221437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.221403 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr"] Nov 04 00:00:01.223767 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:00:01.223727 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5537b60_f90f_47fd_aac1_433ec053333a.slice/crio-f49741805580c1202f81b5c3bca43e13c4294fccf5b6f9c19d7a453678589161 WatchSource:0}: Error finding container f49741805580c1202f81b5c3bca43e13c4294fccf5b6f9c19d7a453678589161: Status 404 returned error can't find the container with id f49741805580c1202f81b5c3bca43e13c4294fccf5b6f9c19d7a453678589161 Nov 04 00:00:01.336387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.336348 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" event={"ID":"a5537b60-f90f-47fd-aac1-433ec053333a","Type":"ContainerStarted","Data":"f49741805580c1202f81b5c3bca43e13c4294fccf5b6f9c19d7a453678589161"} Nov 04 00:00:01.337770 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.337735 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" event={"ID":"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3","Type":"ContainerStarted","Data":"1c41379c776dbc323da11690948d5f574224fe1ea868e19425fed24a928281d2"} Nov 04 00:00:01.339192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.339159 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" event={"ID":"95df9311-7c2c-48b1-b9df-ef7632b87034","Type":"ContainerStarted","Data":"efd90b531b3c6e573430f5fa2c96e3f27ad2269fba4ca57ae2f8f96cf3825983"} Nov 04 00:00:01.340505 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.340484 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" event={"ID":"50f57d3c-6e1a-4cb2-82d6-969494d04963","Type":"ContainerStarted","Data":"3eb759b2aad68d7216f2facaff6bcfbf2ef9c1b868f9df1dc68e3baf44a5a78f"} Nov 04 00:00:01.341886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.341863 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" event={"ID":"5a430fb1-1948-4357-87e9-58af7f08b30b","Type":"ContainerStarted","Data":"43e603ff1d77f024298721f7335d21856aba6519157251fda24e1e86fc64eb45"} Nov 04 00:00:01.827110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:01.827070 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:02.352602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.352513 6208 generic.go:356] "Generic (PLEG): container finished" podID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerID="48f91d2f1fce6831553a61ed42354fde5940f8241e014bdb7ce032d7a04cfa26" exitCode=60 Nov 04 00:00:02.353017 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.352595 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" event={"ID":"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3","Type":"ContainerDied","Data":"48f91d2f1fce6831553a61ed42354fde5940f8241e014bdb7ce032d7a04cfa26"} Nov 04 00:00:02.353017 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.352935 6208 scope.go:122] "RemoveContainer" containerID="48f91d2f1fce6831553a61ed42354fde5940f8241e014bdb7ce032d7a04cfa26" Nov 04 00:00:02.355021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.354958 6208 generic.go:356] "Generic (PLEG): container finished" podID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerID="48e611ac7df02444d60638952607a1521fb28496e04a7ba31abeabf213799105" exitCode=60 Nov 04 00:00:02.355158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.355064 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" event={"ID":"95df9311-7c2c-48b1-b9df-ef7632b87034","Type":"ContainerDied","Data":"48e611ac7df02444d60638952607a1521fb28496e04a7ba31abeabf213799105"} Nov 04 00:00:02.355314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.355297 6208 scope.go:122] "RemoveContainer" containerID="48e611ac7df02444d60638952607a1521fb28496e04a7ba31abeabf213799105" Nov 04 00:00:02.357243 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.357220 6208 generic.go:356] "Generic (PLEG): container finished" podID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerID="3dd25f1a92070d5b3c2ee80398b60ef837ec4f631b6aac57abc212d0735efb55" exitCode=60 Nov 04 00:00:02.357355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.357322 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" event={"ID":"50f57d3c-6e1a-4cb2-82d6-969494d04963","Type":"ContainerDied","Data":"3dd25f1a92070d5b3c2ee80398b60ef837ec4f631b6aac57abc212d0735efb55"} Nov 04 00:00:02.357574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.357522 6208 scope.go:122] "RemoveContainer" containerID="3dd25f1a92070d5b3c2ee80398b60ef837ec4f631b6aac57abc212d0735efb55" Nov 04 00:00:02.359136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.359113 6208 generic.go:356] "Generic (PLEG): container finished" podID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerID="744f32a81ea566d7e0c92df67908f8d91d7b495a2b1f47ac60cb6544e1374c91" exitCode=60 Nov 04 00:00:02.359245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.359138 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" event={"ID":"5a430fb1-1948-4357-87e9-58af7f08b30b","Type":"ContainerDied","Data":"744f32a81ea566d7e0c92df67908f8d91d7b495a2b1f47ac60cb6544e1374c91"} Nov 04 00:00:02.359410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.359391 6208 scope.go:122] "RemoveContainer" containerID="744f32a81ea566d7e0c92df67908f8d91d7b495a2b1f47ac60cb6544e1374c91" Nov 04 00:00:02.361091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.361064 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" event={"ID":"a5537b60-f90f-47fd-aac1-433ec053333a","Type":"ContainerStarted","Data":"66d113a12e819455b089158dfadc544be6e1e0988ce7eb38fc20fdb4fe568b90"} Nov 04 00:00:02.429547 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.429500 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" podStartSLOduration=2.42948594 podStartE2EDuration="2.42948594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:02.428157226 +0000 UTC m=+1821.250753400" watchObservedRunningTime="2026-11-04 00:00:02.42948594 +0000 UTC m=+1821.252082103" Nov 04 00:00:02.638996 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:02.638899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:03.371810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.371775 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" event={"ID":"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3","Type":"ContainerStarted","Data":"638d84f4a45391166bd6a4b89fd1326f99b285bbaf90450677ae2c9d288b5b0c"} Nov 04 00:00:03.373805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.373781 6208 generic.go:356] "Generic (PLEG): container finished" podID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerID="501fed128607cdc4b40d657a851651d5fb4c0999fda253e3fe39af7a6ce0c7fd" exitCode=60 Nov 04 00:00:03.373924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.373860 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" event={"ID":"95df9311-7c2c-48b1-b9df-ef7632b87034","Type":"ContainerDied","Data":"501fed128607cdc4b40d657a851651d5fb4c0999fda253e3fe39af7a6ce0c7fd"} Nov 04 00:00:03.373924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.373918 6208 scope.go:122] "RemoveContainer" containerID="48e611ac7df02444d60638952607a1521fb28496e04a7ba31abeabf213799105" Nov 04 00:00:03.374279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.374263 6208 scope.go:122] "RemoveContainer" containerID="501fed128607cdc4b40d657a851651d5fb4c0999fda253e3fe39af7a6ce0c7fd" Nov 04 00:00:03.374598 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:03.374579 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh_odh-ai-gateway-infra(95df9311-7c2c-48b1-b9df-ef7632b87034)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" Nov 04 00:00:03.376797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.376771 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" event={"ID":"50f57d3c-6e1a-4cb2-82d6-969494d04963","Type":"ContainerStarted","Data":"9209b3740c33f3568298b21b2d6b82981f2ffef10ee99f8725fb2dfca62792f7"} Nov 04 00:00:03.379228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.379201 6208 generic.go:356] "Generic (PLEG): container finished" podID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerID="daee44da02add39caf2c1acf6026548175f667b144dcfbc587939f8dc6ce7c6c" exitCode=60 Nov 04 00:00:03.379349 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.379282 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" event={"ID":"5a430fb1-1948-4357-87e9-58af7f08b30b","Type":"ContainerDied","Data":"daee44da02add39caf2c1acf6026548175f667b144dcfbc587939f8dc6ce7c6c"} Nov 04 00:00:03.379490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.379476 6208 scope.go:122] "RemoveContainer" containerID="daee44da02add39caf2c1acf6026548175f667b144dcfbc587939f8dc6ce7c6c" Nov 04 00:00:03.379761 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:03.379739 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895840-rdbs2_odh-ai-gateway-infra(5a430fb1-1948-4357-87e9-58af7f08b30b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" Nov 04 00:00:03.380966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.380942 6208 generic.go:356] "Generic (PLEG): container finished" podID="a5537b60-f90f-47fd-aac1-433ec053333a" containerID="66d113a12e819455b089158dfadc544be6e1e0988ce7eb38fc20fdb4fe568b90" exitCode=60 Nov 04 00:00:03.381094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.380976 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" event={"ID":"a5537b60-f90f-47fd-aac1-433ec053333a","Type":"ContainerDied","Data":"66d113a12e819455b089158dfadc544be6e1e0988ce7eb38fc20fdb4fe568b90"} Nov 04 00:00:03.381252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.381236 6208 scope.go:122] "RemoveContainer" containerID="66d113a12e819455b089158dfadc544be6e1e0988ce7eb38fc20fdb4fe568b90" Nov 04 00:00:03.391447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.391395 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" podStartSLOduration=3.391379018 podStartE2EDuration="3.391379018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:03.389420955 +0000 UTC m=+1822.212017118" watchObservedRunningTime="2026-11-04 00:00:03.391379018 +0000 UTC m=+1822.213975183" Nov 04 00:00:03.407706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.407658 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" podStartSLOduration=3.407640609 podStartE2EDuration="3.407640609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:03.406936746 +0000 UTC m=+1822.229532910" watchObservedRunningTime="2026-11-04 00:00:03.407640609 +0000 UTC m=+1822.230236771" Nov 04 00:00:03.550232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.550201 6208 scope.go:122] "RemoveContainer" containerID="744f32a81ea566d7e0c92df67908f8d91d7b495a2b1f47ac60cb6544e1374c91" Nov 04 00:00:03.645271 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:03.645239 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:04.390145 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.390113 6208 scope.go:122] "RemoveContainer" containerID="daee44da02add39caf2c1acf6026548175f667b144dcfbc587939f8dc6ce7c6c" Nov 04 00:00:04.390614 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:04.390523 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895840-rdbs2_odh-ai-gateway-infra(5a430fb1-1948-4357-87e9-58af7f08b30b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" Nov 04 00:00:04.391854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.391825 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" event={"ID":"a5537b60-f90f-47fd-aac1-433ec053333a","Type":"ContainerStarted","Data":"04b2100abf897743430132594c84c0444e26572b414d3bdaf0d061093a148be2"} Nov 04 00:00:04.393978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.393950 6208 generic.go:356] "Generic (PLEG): container finished" podID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerID="638d84f4a45391166bd6a4b89fd1326f99b285bbaf90450677ae2c9d288b5b0c" exitCode=60 Nov 04 00:00:04.394134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.394027 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" event={"ID":"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3","Type":"ContainerDied","Data":"638d84f4a45391166bd6a4b89fd1326f99b285bbaf90450677ae2c9d288b5b0c"} Nov 04 00:00:04.394134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.394111 6208 scope.go:122] "RemoveContainer" containerID="48f91d2f1fce6831553a61ed42354fde5940f8241e014bdb7ce032d7a04cfa26" Nov 04 00:00:04.394336 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.394318 6208 scope.go:122] "RemoveContainer" containerID="638d84f4a45391166bd6a4b89fd1326f99b285bbaf90450677ae2c9d288b5b0c" Nov 04 00:00:04.394701 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:04.394680 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk_odh-ai-gateway-infra(b26dbe69-425e-4f9e-8758-4d86d7d2aeb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" Nov 04 00:00:04.396131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.396111 6208 scope.go:122] "RemoveContainer" containerID="501fed128607cdc4b40d657a851651d5fb4c0999fda253e3fe39af7a6ce0c7fd" Nov 04 00:00:04.396422 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:04.396404 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh_odh-ai-gateway-infra(95df9311-7c2c-48b1-b9df-ef7632b87034)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" Nov 04 00:00:04.397786 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.397766 6208 generic.go:356] "Generic (PLEG): container finished" podID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerID="9209b3740c33f3568298b21b2d6b82981f2ffef10ee99f8725fb2dfca62792f7" exitCode=60 Nov 04 00:00:04.397867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.397830 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" event={"ID":"50f57d3c-6e1a-4cb2-82d6-969494d04963","Type":"ContainerDied","Data":"9209b3740c33f3568298b21b2d6b82981f2ffef10ee99f8725fb2dfca62792f7"} Nov 04 00:00:04.398073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.398018 6208 scope.go:122] "RemoveContainer" containerID="9209b3740c33f3568298b21b2d6b82981f2ffef10ee99f8725fb2dfca62792f7" Nov 04 00:00:04.398338 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:04.398314 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2_odh-ai-gateway-infra(50f57d3c-6e1a-4cb2-82d6-969494d04963)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" Nov 04 00:00:04.408488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.408467 6208 scope.go:122] "RemoveContainer" containerID="3dd25f1a92070d5b3c2ee80398b60ef837ec4f631b6aac57abc212d0735efb55" Nov 04 00:00:04.639878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:04.639841 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:05.407618 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:05.407585 6208 generic.go:356] "Generic (PLEG): container finished" podID="a5537b60-f90f-47fd-aac1-433ec053333a" containerID="04b2100abf897743430132594c84c0444e26572b414d3bdaf0d061093a148be2" exitCode=60 Nov 04 00:00:05.408087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:05.407692 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" event={"ID":"a5537b60-f90f-47fd-aac1-433ec053333a","Type":"ContainerDied","Data":"04b2100abf897743430132594c84c0444e26572b414d3bdaf0d061093a148be2"} Nov 04 00:00:05.408087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:05.407753 6208 scope.go:122] "RemoveContainer" containerID="66d113a12e819455b089158dfadc544be6e1e0988ce7eb38fc20fdb4fe568b90" Nov 04 00:00:05.408087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:05.407978 6208 scope.go:122] "RemoveContainer" containerID="04b2100abf897743430132594c84c0444e26572b414d3bdaf0d061093a148be2" Nov 04 00:00:05.408473 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:05.408447 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr_odh-ai-gateway-infra(a5537b60-f90f-47fd-aac1-433ec053333a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" Nov 04 00:00:05.409860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:05.409838 6208 scope.go:122] "RemoveContainer" containerID="638d84f4a45391166bd6a4b89fd1326f99b285bbaf90450677ae2c9d288b5b0c" Nov 04 00:00:05.410194 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:05.410175 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk_odh-ai-gateway-infra(b26dbe69-425e-4f9e-8758-4d86d7d2aeb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" Nov 04 00:00:05.411626 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:05.411604 6208 scope.go:122] "RemoveContainer" containerID="9209b3740c33f3568298b21b2d6b82981f2ffef10ee99f8725fb2dfca62792f7" Nov 04 00:00:05.411959 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:05.411938 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2_odh-ai-gateway-infra(50f57d3c-6e1a-4cb2-82d6-969494d04963)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" Nov 04 00:00:05.650053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:05.649986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:05.733141 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:05.732943 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:00:06.420023 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:06.419999 6208 scope.go:122] "RemoveContainer" containerID="04b2100abf897743430132594c84c0444e26572b414d3bdaf0d061093a148be2" Nov 04 00:00:06.420462 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:06.420349 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr_odh-ai-gateway-infra(a5537b60-f90f-47fd-aac1-433ec053333a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" Nov 04 00:00:06.447509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:06.447475 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:06.565047 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:06.565000 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:07.187613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:07.187577 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:07.903338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:07.903302 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:08.622098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:08.622063 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:09.451932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:09.451898 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:10.237228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:10.237192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:11.138252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:11.138216 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:11.655322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:11.655285 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:11.944444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:11.944351 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:12.804312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:12.804268 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:13.640017 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:13.639984 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:14.287087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:14.287059 6208 scope.go:122] "RemoveContainer" containerID="501fed128607cdc4b40d657a851651d5fb4c0999fda253e3fe39af7a6ce0c7fd" Nov 04 00:00:14.623486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:14.623444 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:15.298151 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:15.298119 6208 scope.go:122] "RemoveContainer" containerID="daee44da02add39caf2c1acf6026548175f667b144dcfbc587939f8dc6ce7c6c" Nov 04 00:00:15.487848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:15.487818 6208 generic.go:356] "Generic (PLEG): container finished" podID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerID="a910459aa59bd3b5088f5fc3fdb6ccda840ff3bf1b7a3946d3826f4e3e0923a4" exitCode=60 Nov 04 00:00:15.488011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:15.487895 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" event={"ID":"95df9311-7c2c-48b1-b9df-ef7632b87034","Type":"ContainerDied","Data":"a910459aa59bd3b5088f5fc3fdb6ccda840ff3bf1b7a3946d3826f4e3e0923a4"} Nov 04 00:00:15.488011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:15.487968 6208 scope.go:122] "RemoveContainer" containerID="501fed128607cdc4b40d657a851651d5fb4c0999fda253e3fe39af7a6ce0c7fd" Nov 04 00:00:15.488445 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:15.488426 6208 scope.go:122] "RemoveContainer" containerID="a910459aa59bd3b5088f5fc3fdb6ccda840ff3bf1b7a3946d3826f4e3e0923a4" Nov 04 00:00:15.488839 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:15.488814 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh_odh-ai-gateway-infra(95df9311-7c2c-48b1-b9df-ef7632b87034)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" Nov 04 00:00:15.528538 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:15.528508 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:15.720698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:15.720661 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:16.287458 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.287429 6208 scope.go:122] "RemoveContainer" containerID="04b2100abf897743430132594c84c0444e26572b414d3bdaf0d061093a148be2" Nov 04 00:00:16.498248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.498215 6208 generic.go:356] "Generic (PLEG): container finished" podID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerID="957510acc03b94c77e5e1175593fee0495aa13f86eb0ac4e934e514d6d9e81bd" exitCode=60 Nov 04 00:00:16.498642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.498283 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" event={"ID":"5a430fb1-1948-4357-87e9-58af7f08b30b","Type":"ContainerDied","Data":"957510acc03b94c77e5e1175593fee0495aa13f86eb0ac4e934e514d6d9e81bd"} Nov 04 00:00:16.498642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.498337 6208 scope.go:122] "RemoveContainer" containerID="daee44da02add39caf2c1acf6026548175f667b144dcfbc587939f8dc6ce7c6c" Nov 04 00:00:16.499421 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.499396 6208 scope.go:122] "RemoveContainer" containerID="957510acc03b94c77e5e1175593fee0495aa13f86eb0ac4e934e514d6d9e81bd" Nov 04 00:00:16.504105 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:16.500114 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895840-rdbs2_odh-ai-gateway-infra(5a430fb1-1948-4357-87e9-58af7f08b30b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" Nov 04 00:00:16.528850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.528826 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh"] Nov 04 00:00:16.550367 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.550339 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:16.738966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.738940 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:16.831498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.831413 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95df9311-7c2c-48b1-b9df-ef7632b87034-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df9311-7c2c-48b1-b9df-ef7632b87034-maas-api-service-ca\") pod \"95df9311-7c2c-48b1-b9df-ef7632b87034\" (UID: \"95df9311-7c2c-48b1-b9df-ef7632b87034\") " Nov 04 00:00:16.831657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.831545 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95df9311-7c2c-48b1-b9df-ef7632b87034-kube-api-access-zjlkn\" (UniqueName: \"kubernetes.io/projected/95df9311-7c2c-48b1-b9df-ef7632b87034-kube-api-access-zjlkn\") pod \"95df9311-7c2c-48b1-b9df-ef7632b87034\" (UID: \"95df9311-7c2c-48b1-b9df-ef7632b87034\") " Nov 04 00:00:16.831792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.831768 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95df9311-7c2c-48b1-b9df-ef7632b87034-maas-api-service-ca" pod "95df9311-7c2c-48b1-b9df-ef7632b87034" (UID: "95df9311-7c2c-48b1-b9df-ef7632b87034"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:00:16.831984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.831965 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df9311-7c2c-48b1-b9df-ef7632b87034-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:16.833725 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.833699 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95df9311-7c2c-48b1-b9df-ef7632b87034-kube-api-access-zjlkn" pod "95df9311-7c2c-48b1-b9df-ef7632b87034" (UID: "95df9311-7c2c-48b1-b9df-ef7632b87034"). InnerVolumeSpecName "kube-api-access-zjlkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:00:16.933004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:16.932962 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjlkn\" (UniqueName: \"kubernetes.io/projected/95df9311-7c2c-48b1-b9df-ef7632b87034-kube-api-access-zjlkn\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:17.287837 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.287811 6208 scope.go:122] "RemoveContainer" containerID="9209b3740c33f3568298b21b2d6b82981f2ffef10ee99f8725fb2dfca62792f7" Nov 04 00:00:17.287986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.287968 6208 scope.go:122] "RemoveContainer" containerID="638d84f4a45391166bd6a4b89fd1326f99b285bbaf90450677ae2c9d288b5b0c" Nov 04 00:00:17.520315 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.520227 6208 generic.go:356] "Generic (PLEG): container finished" podID="a5537b60-f90f-47fd-aac1-433ec053333a" containerID="f3f2baf15d439d70d600a838bbd1a8eabfd73eb93db5368e370323e39f3c91e9" exitCode=60 Nov 04 00:00:17.520764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.520302 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" event={"ID":"a5537b60-f90f-47fd-aac1-433ec053333a","Type":"ContainerDied","Data":"f3f2baf15d439d70d600a838bbd1a8eabfd73eb93db5368e370323e39f3c91e9"} Nov 04 00:00:17.520764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.520372 6208 scope.go:122] "RemoveContainer" containerID="04b2100abf897743430132594c84c0444e26572b414d3bdaf0d061093a148be2" Nov 04 00:00:17.520903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.520832 6208 scope.go:122] "RemoveContainer" containerID="f3f2baf15d439d70d600a838bbd1a8eabfd73eb93db5368e370323e39f3c91e9" Nov 04 00:00:17.521274 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:17.521245 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr_odh-ai-gateway-infra(a5537b60-f90f-47fd-aac1-433ec053333a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" Nov 04 00:00:17.522455 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.522432 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" Nov 04 00:00:17.522563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.522485 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh" event={"ID":"95df9311-7c2c-48b1-b9df-ef7632b87034","Type":"ContainerDied","Data":"efd90b531b3c6e573430f5fa2c96e3f27ad2269fba4ca57ae2f8f96cf3825983"} Nov 04 00:00:17.537589 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.537569 6208 scope.go:122] "RemoveContainer" containerID="a910459aa59bd3b5088f5fc3fdb6ccda840ff3bf1b7a3946d3826f4e3e0923a4" Nov 04 00:00:17.558260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.558234 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2"] Nov 04 00:00:17.583276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.583193 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh"] Nov 04 00:00:17.592054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.591823 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895840-4fmfh"] Nov 04 00:00:17.783103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.781341 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:17.815649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.814853 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:17.975680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.975652 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a430fb1-1948-4357-87e9-58af7f08b30b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a430fb1-1948-4357-87e9-58af7f08b30b-maas-api-service-ca\") pod \"5a430fb1-1948-4357-87e9-58af7f08b30b\" (UID: \"5a430fb1-1948-4357-87e9-58af7f08b30b\") " Nov 04 00:00:17.975814 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.975769 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a430fb1-1948-4357-87e9-58af7f08b30b-kube-api-access-jpk5k\" (UniqueName: \"kubernetes.io/projected/5a430fb1-1948-4357-87e9-58af7f08b30b-kube-api-access-jpk5k\") pod \"5a430fb1-1948-4357-87e9-58af7f08b30b\" (UID: \"5a430fb1-1948-4357-87e9-58af7f08b30b\") " Nov 04 00:00:17.976074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.976051 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a430fb1-1948-4357-87e9-58af7f08b30b-maas-api-service-ca" pod "5a430fb1-1948-4357-87e9-58af7f08b30b" (UID: "5a430fb1-1948-4357-87e9-58af7f08b30b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:00:17.976526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.976497 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a430fb1-1948-4357-87e9-58af7f08b30b-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:17.977646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:17.977618 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a430fb1-1948-4357-87e9-58af7f08b30b-kube-api-access-jpk5k" pod "5a430fb1-1948-4357-87e9-58af7f08b30b" (UID: "5a430fb1-1948-4357-87e9-58af7f08b30b"). InnerVolumeSpecName "kube-api-access-jpk5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:00:18.078581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.078500 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpk5k\" (UniqueName: \"kubernetes.io/projected/5a430fb1-1948-4357-87e9-58af7f08b30b-kube-api-access-jpk5k\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:18.532430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.532404 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" Nov 04 00:00:18.532860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.532422 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2" event={"ID":"5a430fb1-1948-4357-87e9-58af7f08b30b","Type":"ContainerDied","Data":"43e603ff1d77f024298721f7335d21856aba6519157251fda24e1e86fc64eb45"} Nov 04 00:00:18.532860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.532488 6208 scope.go:122] "RemoveContainer" containerID="957510acc03b94c77e5e1175593fee0495aa13f86eb0ac4e934e514d6d9e81bd" Nov 04 00:00:18.536363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.536338 6208 generic.go:356] "Generic (PLEG): container finished" podID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerID="8052d808be05e314da9be9105724cda9f70f2ced34d6972ff9b6bbb9c1bee5d3" exitCode=60 Nov 04 00:00:18.536524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.536405 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" event={"ID":"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3","Type":"ContainerDied","Data":"8052d808be05e314da9be9105724cda9f70f2ced34d6972ff9b6bbb9c1bee5d3"} Nov 04 00:00:18.536887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.536846 6208 scope.go:122] "RemoveContainer" containerID="8052d808be05e314da9be9105724cda9f70f2ced34d6972ff9b6bbb9c1bee5d3" Nov 04 00:00:18.537298 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:18.537263 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk_odh-ai-gateway-infra(b26dbe69-425e-4f9e-8758-4d86d7d2aeb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" Nov 04 00:00:18.540016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.539997 6208 generic.go:356] "Generic (PLEG): container finished" podID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerID="27cd48fd6036f76928101116565215c53075b45dd585053bb68f60866994dcc8" exitCode=60 Nov 04 00:00:18.540129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.540077 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" event={"ID":"50f57d3c-6e1a-4cb2-82d6-969494d04963","Type":"ContainerDied","Data":"27cd48fd6036f76928101116565215c53075b45dd585053bb68f60866994dcc8"} Nov 04 00:00:18.540493 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.540475 6208 scope.go:122] "RemoveContainer" containerID="27cd48fd6036f76928101116565215c53075b45dd585053bb68f60866994dcc8" Nov 04 00:00:18.540840 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:18.540810 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2_odh-ai-gateway-infra(50f57d3c-6e1a-4cb2-82d6-969494d04963)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" Nov 04 00:00:18.545384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.545363 6208 scope.go:122] "RemoveContainer" containerID="638d84f4a45391166bd6a4b89fd1326f99b285bbaf90450677ae2c9d288b5b0c" Nov 04 00:00:18.560792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.560763 6208 scope.go:122] "RemoveContainer" containerID="9209b3740c33f3568298b21b2d6b82981f2ffef10ee99f8725fb2dfca62792f7" Nov 04 00:00:18.573234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.573209 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr"] Nov 04 00:00:18.636310 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.636115 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2"] Nov 04 00:00:18.644343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.644311 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895840-rdbs2"] Nov 04 00:00:18.701094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.699185 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:18.819531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.819496 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:18.822167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.822008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:18.902300 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.902210 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5537b60-f90f-47fd-aac1-433ec053333a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5537b60-f90f-47fd-aac1-433ec053333a-maas-api-service-ca\") pod \"a5537b60-f90f-47fd-aac1-433ec053333a\" (UID: \"a5537b60-f90f-47fd-aac1-433ec053333a\") " Nov 04 00:00:18.902544 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.902519 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5537b60-f90f-47fd-aac1-433ec053333a-kube-api-access-7c5mk\" (UniqueName: \"kubernetes.io/projected/a5537b60-f90f-47fd-aac1-433ec053333a-kube-api-access-7c5mk\") pod \"a5537b60-f90f-47fd-aac1-433ec053333a\" (UID: \"a5537b60-f90f-47fd-aac1-433ec053333a\") " Nov 04 00:00:18.902644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.902608 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5537b60-f90f-47fd-aac1-433ec053333a-maas-api-service-ca" pod "a5537b60-f90f-47fd-aac1-433ec053333a" (UID: "a5537b60-f90f-47fd-aac1-433ec053333a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:00:18.903232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.903207 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5537b60-f90f-47fd-aac1-433ec053333a-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:18.904871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:18.904840 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5537b60-f90f-47fd-aac1-433ec053333a-kube-api-access-7c5mk" pod "a5537b60-f90f-47fd-aac1-433ec053333a" (UID: "a5537b60-f90f-47fd-aac1-433ec053333a"). InnerVolumeSpecName "kube-api-access-7c5mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:00:19.005121 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.005090 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c5mk\" (UniqueName: \"kubernetes.io/projected/a5537b60-f90f-47fd-aac1-433ec053333a-kube-api-access-7c5mk\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:19.069253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.069215 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l"] Nov 04 00:00:19.070256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070234 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" containerName="cleanup" Nov 04 00:00:19.070317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070260 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" containerName="cleanup" Nov 04 00:00:19.070317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070277 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerName="cleanup" Nov 04 00:00:19.070317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070287 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerName="cleanup" Nov 04 00:00:19.070317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070299 6208 cpu_manager.go:414] "containerMap: removing container" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerName="cleanup" Nov 04 00:00:19.070317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070308 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerName="cleanup" Nov 04 00:00:19.070317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070317 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerName="cleanup" Nov 04 00:00:19.070491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070323 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerName="cleanup" Nov 04 00:00:19.070491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070350 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" containerName="cleanup" Nov 04 00:00:19.070491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070358 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" containerName="cleanup" Nov 04 00:00:19.070491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070374 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" containerName="cleanup" Nov 04 00:00:19.070491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070380 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" containerName="cleanup" Nov 04 00:00:19.070491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070433 6208 cpu_manager.go:414] "containerMap: removing container" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerName="cleanup" Nov 04 00:00:19.070491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070440 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerName="cleanup" Nov 04 00:00:19.070717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070601 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerName="cleanup" Nov 04 00:00:19.070717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070621 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerName="cleanup" Nov 04 00:00:19.070717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070630 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerName="cleanup" Nov 04 00:00:19.070717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070644 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" containerName="cleanup" Nov 04 00:00:19.070717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070653 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerName="cleanup" Nov 04 00:00:19.070717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070662 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerName="cleanup" Nov 04 00:00:19.070717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.070675 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" containerName="cleanup" Nov 04 00:00:19.074922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.074899 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.081327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.081302 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-fa0d0a-openshift-default-dockercfg-6b2s7\"" Nov 04 00:00:19.091788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.091766 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l"] Nov 04 00:00:19.124386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.124341 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:19.224561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.224518 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418b6385-c834-4532-94d5-e65086de1352-istio-podinfo\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.224779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.224623 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-istio-token\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.224779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.224733 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418b6385-c834-4532-94d5-e65086de1352-istiod-ca-cert\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.224779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.224763 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlb7g\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-kube-api-access-qlb7g\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.224967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.224806 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-certs\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.224967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.224831 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-data\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.224967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.224851 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-socket\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.224967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.224932 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-credential-socket\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.225220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.225015 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-envoy\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.270580 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.270455 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:19.319060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.313879 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" path="/var/lib/kubelet/pods/5a430fb1-1948-4357-87e9-58af7f08b30b/volumes" Nov 04 00:00:19.319060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.314649 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" path="/var/lib/kubelet/pods/95df9311-7c2c-48b1-b9df-ef7632b87034/volumes" Nov 04 00:00:19.339873 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.339837 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418b6385-c834-4532-94d5-e65086de1352-istiod-ca-cert\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.340124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.340106 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlb7g\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-kube-api-access-qlb7g\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.340264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.340251 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-certs\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.340369 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.340354 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-data\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.340461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.340449 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-socket\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.340620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.340608 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-credential-socket\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.340774 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.340762 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-envoy\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.340972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.340959 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418b6385-c834-4532-94d5-e65086de1352-istio-podinfo\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.341114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.341100 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-istio-token\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.342267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.342243 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418b6385-c834-4532-94d5-e65086de1352-istiod-ca-cert\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.345236 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.342760 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-socket\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.345415 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.343183 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-certs\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.345537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.343367 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-data\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.345643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.343887 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-credential-socket\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.368698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.368667 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418b6385-c834-4532-94d5-e65086de1352-istio-podinfo\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.369628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.369023 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-envoy\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.376973 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.376946 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-istio-token\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.396500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.396440 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlb7g\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-kube-api-access-qlb7g\") pod \"e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.555198 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.555108 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" Nov 04 00:00:19.555619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.555104 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr" event={"ID":"a5537b60-f90f-47fd-aac1-433ec053333a","Type":"ContainerDied","Data":"f49741805580c1202f81b5c3bca43e13c4294fccf5b6f9c19d7a453678589161"} Nov 04 00:00:19.555619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.555256 6208 scope.go:122] "RemoveContainer" containerID="f3f2baf15d439d70d600a838bbd1a8eabfd73eb93db5368e370323e39f3c91e9" Nov 04 00:00:19.574854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.574818 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk"] Nov 04 00:00:19.579156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.579131 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr"] Nov 04 00:00:19.590813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.590783 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895840-688lr"] Nov 04 00:00:19.607058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.606999 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2"] Nov 04 00:00:19.690598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.690562 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:19.762119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.762004 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:19.838882 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.838839 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:19.851653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.851624 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-maas-api-service-ca\") pod \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\" (UID: \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\") " Nov 04 00:00:19.851780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.851710 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f57d3c-6e1a-4cb2-82d6-969494d04963-kube-api-access-cbhzj\" (UniqueName: \"kubernetes.io/projected/50f57d3c-6e1a-4cb2-82d6-969494d04963-kube-api-access-cbhzj\") pod \"50f57d3c-6e1a-4cb2-82d6-969494d04963\" (UID: \"50f57d3c-6e1a-4cb2-82d6-969494d04963\") " Nov 04 00:00:19.851843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.851816 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f57d3c-6e1a-4cb2-82d6-969494d04963-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f57d3c-6e1a-4cb2-82d6-969494d04963-maas-api-service-ca\") pod \"50f57d3c-6e1a-4cb2-82d6-969494d04963\" (UID: \"50f57d3c-6e1a-4cb2-82d6-969494d04963\") " Nov 04 00:00:19.851993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.851971 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-kube-api-access-wmhff\" (UniqueName: \"kubernetes.io/projected/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-kube-api-access-wmhff\") pod \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\" (UID: \"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3\") " Nov 04 00:00:19.852389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.852364 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-maas-api-service-ca" pod "b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" (UID: "b26dbe69-425e-4f9e-8758-4d86d7d2aeb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:00:19.852495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.852388 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f57d3c-6e1a-4cb2-82d6-969494d04963-maas-api-service-ca" pod "50f57d3c-6e1a-4cb2-82d6-969494d04963" (UID: "50f57d3c-6e1a-4cb2-82d6-969494d04963"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:00:19.852751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.852732 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:19.852832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.852754 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f57d3c-6e1a-4cb2-82d6-969494d04963-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:19.856527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.854352 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-kube-api-access-wmhff" pod "b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" (UID: "b26dbe69-425e-4f9e-8758-4d86d7d2aeb3"). InnerVolumeSpecName "kube-api-access-wmhff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:00:19.856527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.854380 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f57d3c-6e1a-4cb2-82d6-969494d04963-kube-api-access-cbhzj" pod "50f57d3c-6e1a-4cb2-82d6-969494d04963" (UID: "50f57d3c-6e1a-4cb2-82d6-969494d04963"). InnerVolumeSpecName "kube-api-access-cbhzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:00:19.953883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.953853 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbhzj\" (UniqueName: \"kubernetes.io/projected/50f57d3c-6e1a-4cb2-82d6-969494d04963-kube-api-access-cbhzj\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:19.953883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:19.953879 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhff\" (UniqueName: \"kubernetes.io/projected/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3-kube-api-access-wmhff\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:20.026257 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.026213 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l"] Nov 04 00:00:20.027819 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:00:20.027773 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418b6385_c834_4532_94d5_e65086de1352.slice/crio-0323e1cf51f030f0777ef03b92d2bf72f1481c0b5331bdf13305da05888fea43 WatchSource:0}: Error finding container 0323e1cf51f030f0777ef03b92d2bf72f1481c0b5331bdf13305da05888fea43: Status 404 returned error can't find the container with id 0323e1cf51f030f0777ef03b92d2bf72f1481c0b5331bdf13305da05888fea43 Nov 04 00:00:20.031101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.031023 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:00:20.031205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.031147 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:00:20.031205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.031197 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:00:20.164944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.164911 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:20.566067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.566006 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" event={"ID":"418b6385-c834-4532-94d5-e65086de1352","Type":"ContainerStarted","Data":"6f2e4ee658d9b0c43dce210c1c2bc7954ce1b72b510cb3bb236892142a49605f"} Nov 04 00:00:20.566538 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.566098 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" event={"ID":"418b6385-c834-4532-94d5-e65086de1352","Type":"ContainerStarted","Data":"0323e1cf51f030f0777ef03b92d2bf72f1481c0b5331bdf13305da05888fea43"} Nov 04 00:00:20.567505 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.567480 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" Nov 04 00:00:20.567645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.567514 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk" event={"ID":"b26dbe69-425e-4f9e-8758-4d86d7d2aeb3","Type":"ContainerDied","Data":"1c41379c776dbc323da11690948d5f574224fe1ea868e19425fed24a928281d2"} Nov 04 00:00:20.567645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.567576 6208 scope.go:122] "RemoveContainer" containerID="8052d808be05e314da9be9105724cda9f70f2ced34d6972ff9b6bbb9c1bee5d3" Nov 04 00:00:20.569379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.569352 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" event={"ID":"50f57d3c-6e1a-4cb2-82d6-969494d04963","Type":"ContainerDied","Data":"3eb759b2aad68d7216f2facaff6bcfbf2ef9c1b868f9df1dc68e3baf44a5a78f"} Nov 04 00:00:20.569458 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.569377 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2" Nov 04 00:00:20.580221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.580205 6208 scope.go:122] "RemoveContainer" containerID="27cd48fd6036f76928101116565215c53075b45dd585053bb68f60866994dcc8" Nov 04 00:00:20.601800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.601752 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" podStartSLOduration=1.60173572 podStartE2EDuration="1.60173572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:20.599802311 +0000 UTC m=+1839.422398497" watchObservedRunningTime="2026-11-04 00:00:20.60173572 +0000 UTC m=+1839.424331885" Nov 04 00:00:20.618548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.618480 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2"] Nov 04 00:00:20.622563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.622541 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-58b75d-29895840-2lqj2"] Nov 04 00:00:20.634939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.634915 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk"] Nov 04 00:00:20.641099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.641074 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5738a9-29895840-xkdnk"] Nov 04 00:00:20.691615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:20.691588 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:21.126583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:21.126544 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:21.298199 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:21.298167 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" path="/var/lib/kubelet/pods/50f57d3c-6e1a-4cb2-82d6-969494d04963/volumes" Nov 04 00:00:21.298699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:21.298681 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" path="/var/lib/kubelet/pods/a5537b60-f90f-47fd-aac1-433ec053333a/volumes" Nov 04 00:00:21.299162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:21.299148 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" path="/var/lib/kubelet/pods/b26dbe69-425e-4f9e-8758-4d86d7d2aeb3/volumes" Nov 04 00:00:21.697928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:21.697897 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:22.381129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:22.381089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:22.586785 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:22.586750 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:22.587903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:22.587864 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:00:23.117332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:23.117298 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:23.293619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:23.293579 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:23.816098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:23.810619 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:24.189212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:24.189107 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:24.322845 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:24.322804 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:25.034815 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:25.034772 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:26.075478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:26.075445 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:26.240963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:26.240925 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:26.855019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:26.854984 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:26.987045 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:26.986997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:27.380559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:27.380519 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:27.547107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:27.547067 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:27.834201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:27.834164 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:28.377845 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:28.377811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:29.072528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:29.072480 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:29.205141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:29.205102 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:29.328337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:29.328239 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:29.443610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:29.443578 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:30.244868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:30.244834 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:31.161685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:31.161650 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:32.166074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:32.166022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:33.021065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:33.020996 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:33.967457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:33.967421 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:34.929000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:34.928960 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:35.712694 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:35.712651 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:00:35.847913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:35.847869 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:36.656025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:36.655991 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:36.765707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:36.765653 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:37.317645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.317609 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:37.420459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.420418 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:37.672562 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.672488 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v"] Nov 04 00:00:37.673269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673243 6208 cpu_manager.go:414] "containerMap: removing container" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerName="cleanup" Nov 04 00:00:37.673269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673266 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673288 6208 cpu_manager.go:414] "containerMap: removing container" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673296 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673318 6208 cpu_manager.go:414] "containerMap: removing container" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673327 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673339 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673345 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a430fb1-1948-4357-87e9-58af7f08b30b" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673366 6208 cpu_manager.go:414] "containerMap: removing container" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673373 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673396 6208 cpu_manager.go:414] "containerMap: removing container" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerName="cleanup" Nov 04 00:00:37.673464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673403 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerName="cleanup" Nov 04 00:00:37.674065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673575 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerName="cleanup" Nov 04 00:00:37.674065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673595 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerName="cleanup" Nov 04 00:00:37.674065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673607 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5537b60-f90f-47fd-aac1-433ec053333a" containerName="cleanup" Nov 04 00:00:37.674065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673618 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerName="cleanup" Nov 04 00:00:37.674065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673632 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerName="cleanup" Nov 04 00:00:37.674065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673645 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="95df9311-7c2c-48b1-b9df-ef7632b87034" containerName="cleanup" Nov 04 00:00:37.674065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.673654 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerName="cleanup" Nov 04 00:00:37.679206 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.679183 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.694934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.691587 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-27f8a1-openshift-default-dockercfg-fqqq2\"" Nov 04 00:00:37.695759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.695625 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:37.703853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.703822 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v"] Nov 04 00:00:37.851019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.847313 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-envoy\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.851019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.847437 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-credential-socket\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.851019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.847488 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-data\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.851019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.847671 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-istio-token\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.851019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.847742 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-socket\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.851019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.847801 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwpx\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-kube-api-access-mgwpx\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.851019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.847908 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676b3179-ba28-4ce1-baa2-d3557794a632-istiod-ca-cert\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.851019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.848083 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676b3179-ba28-4ce1-baa2-d3557794a632-istio-podinfo\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.851019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.848149 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-certs\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.906649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.906610 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:37.952154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.952060 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-credential-socket\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.952430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.952405 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-data\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.972056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.955719 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-istio-token\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.972664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.972641 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-socket\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.972857 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.972839 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwpx\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-kube-api-access-mgwpx\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.973664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.973644 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676b3179-ba28-4ce1-baa2-d3557794a632-istiod-ca-cert\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.982846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.973934 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676b3179-ba28-4ce1-baa2-d3557794a632-istio-podinfo\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.983185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.983155 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-certs\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.983374 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.983354 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-envoy\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.991054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.973235 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-socket\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.991243 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.974899 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676b3179-ba28-4ce1-baa2-d3557794a632-istiod-ca-cert\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.991356 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.952937 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-data\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.991461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.972534 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-credential-socket\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:37.991572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:37.987875 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-certs\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:38.045056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:38.044759 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-istio-token\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:38.049075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:38.045683 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676b3179-ba28-4ce1-baa2-d3557794a632-istio-podinfo\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:38.071055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:38.059141 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-envoy\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:38.071055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:38.060009 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwpx\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-kube-api-access-mgwpx\") pod \"e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:38.109149 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:38.106023 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:38.251174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:38.248667 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:38.382925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:38.382894 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:38.830121 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:38.807940 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 04 00:00:39.105914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:39.105746 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:39.144641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:39.144614 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v"] Nov 04 00:00:39.147601 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:00:39.147569 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676b3179_ba28_4ce1_baa2_d3557794a632.slice/crio-86c39750f43376aef4a623abf3c723b01ba3a5f6362477f5ab41a628f1c0dce3 WatchSource:0}: Error finding container 86c39750f43376aef4a623abf3c723b01ba3a5f6362477f5ab41a628f1c0dce3: Status 404 returned error can't find the container with id 86c39750f43376aef4a623abf3c723b01ba3a5f6362477f5ab41a628f1c0dce3 Nov 04 00:00:39.150538 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:39.150501 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:00:39.150665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:39.150578 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:00:39.150665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:39.150614 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:00:39.222792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:39.222391 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:39.909583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:39.909540 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" event={"ID":"676b3179-ba28-4ce1-baa2-d3557794a632","Type":"ContainerStarted","Data":"12fa0f85e44d5ef6bc2c8288efd2d317a883259b250a36bbca3eb2a42653454d"} Nov 04 00:00:39.909800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:39.909631 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" event={"ID":"676b3179-ba28-4ce1-baa2-d3557794a632","Type":"ContainerStarted","Data":"86c39750f43376aef4a623abf3c723b01ba3a5f6362477f5ab41a628f1c0dce3"} Nov 04 00:00:39.949617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:39.949562 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" podStartSLOduration=2.949545543 podStartE2EDuration="2.949545543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:00:39.936976075 +0000 UTC m=+1858.759572238" watchObservedRunningTime="2026-11-04 00:00:39.949545543 +0000 UTC m=+1858.772141705" Nov 04 00:00:40.172628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:40.172546 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:40.409058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:40.407947 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:40.456816 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:40.456394 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:40.916381 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:40.916296 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:40.917223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:40.917201 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:00:41.043788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:41.043754 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:41.934156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:41.934113 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:42.098878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:42.098843 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:43.043500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:43.043462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:43.918370 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:43.918332 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:44.739257 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:44.739220 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:45.562445 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:45.562415 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:46.335869 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:46.335832 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk"] Nov 04 00:00:46.336424 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:46.336070 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" containerID="cri-o://491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040" gracePeriod=30 Nov 04 00:00:46.380740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:46.380578 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:46.515858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:46.515815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:47.371131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:47.371095 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:48.186914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:48.186874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:49.084598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:49.084560 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:49.199633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:49.199598 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:49.310090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:49.310029 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:49.593293 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:49.593257 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:49.875558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:49.875465 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:50.214706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:50.214675 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:50.826480 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:50.826444 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:51.195295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.195210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:51.310694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.310653 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:51.436524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.436490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:51.538008 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.537978 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:51.729799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.729770 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 04 00:00:51.822362 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822266 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-podinfo\") pod \"29e697bb-1d96-4ee6-aa78-78afce02e434\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " Nov 04 00:00:51.822362 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822332 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-credential-socket\") pod \"29e697bb-1d96-4ee6-aa78-78afce02e434\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " Nov 04 00:00:51.822648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822447 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e697bb-1d96-4ee6-aa78-78afce02e434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e697bb-1d96-4ee6-aa78-78afce02e434-istiod-ca-cert\") pod \"29e697bb-1d96-4ee6-aa78-78afce02e434\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " Nov 04 00:00:51.822648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822584 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-envoy\") pod \"29e697bb-1d96-4ee6-aa78-78afce02e434\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " Nov 04 00:00:51.822769 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822649 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-token\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-token\") pod \"29e697bb-1d96-4ee6-aa78-78afce02e434\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " Nov 04 00:00:51.822769 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822722 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-credential-socket" pod "29e697bb-1d96-4ee6-aa78-78afce02e434" (UID: "29e697bb-1d96-4ee6-aa78-78afce02e434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:00:51.822893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822775 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-socket\") pod \"29e697bb-1d96-4ee6-aa78-78afce02e434\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " Nov 04 00:00:51.822893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822850 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-kube-api-access-q59tt\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-kube-api-access-q59tt\") pod \"29e697bb-1d96-4ee6-aa78-78afce02e434\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " Nov 04 00:00:51.822893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822886 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-certs\") pod \"29e697bb-1d96-4ee6-aa78-78afce02e434\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " Nov 04 00:00:51.823069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.822935 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-data\") pod \"29e697bb-1d96-4ee6-aa78-78afce02e434\" (UID: \"29e697bb-1d96-4ee6-aa78-78afce02e434\") " Nov 04 00:00:51.823132 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.823062 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e697bb-1d96-4ee6-aa78-78afce02e434-istiod-ca-cert" pod "29e697bb-1d96-4ee6-aa78-78afce02e434" (UID: "29e697bb-1d96-4ee6-aa78-78afce02e434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:00:51.823517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.823314 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-socket" pod "29e697bb-1d96-4ee6-aa78-78afce02e434" (UID: "29e697bb-1d96-4ee6-aa78-78afce02e434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:00:51.823644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.823553 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-data" pod "29e697bb-1d96-4ee6-aa78-78afce02e434" (UID: "29e697bb-1d96-4ee6-aa78-78afce02e434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:00:51.823975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.823780 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-certs" pod "29e697bb-1d96-4ee6-aa78-78afce02e434" (UID: "29e697bb-1d96-4ee6-aa78-78afce02e434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:00:51.823975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.823896 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:51.823975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.823920 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:51.823975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.823936 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:51.823975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.823949 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e697bb-1d96-4ee6-aa78-78afce02e434-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:51.825821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.825781 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-envoy" pod "29e697bb-1d96-4ee6-aa78-78afce02e434" (UID: "29e697bb-1d96-4ee6-aa78-78afce02e434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:00:51.825821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.825792 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-podinfo" pod "29e697bb-1d96-4ee6-aa78-78afce02e434" (UID: "29e697bb-1d96-4ee6-aa78-78afce02e434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:00:51.825983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.825918 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-token" pod "29e697bb-1d96-4ee6-aa78-78afce02e434" (UID: "29e697bb-1d96-4ee6-aa78-78afce02e434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:00:51.826095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.826074 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-kube-api-access-q59tt" pod "29e697bb-1d96-4ee6-aa78-78afce02e434" (UID: "29e697bb-1d96-4ee6-aa78-78afce02e434"). InnerVolumeSpecName "kube-api-access-q59tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:00:51.925714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.925680 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q59tt\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-kube-api-access-q59tt\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:51.925714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.925710 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:51.925714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.925720 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:51.925954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.925730 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:51.925954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:51.925739 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e697bb-1d96-4ee6-aa78-78afce02e434-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:00:52.000182 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.000152 6208 generic.go:356] "Generic (PLEG): container finished" podID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerID="491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040" exitCode=0 Nov 04 00:00:52.000358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.000202 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" Nov 04 00:00:52.000358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.000219 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" event={"ID":"29e697bb-1d96-4ee6-aa78-78afce02e434","Type":"ContainerDied","Data":"491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040"} Nov 04 00:00:52.000358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.000260 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk" event={"ID":"29e697bb-1d96-4ee6-aa78-78afce02e434","Type":"ContainerDied","Data":"9a950dd712ca8f97a509162855419bf4e00d2a32c452232b324fe910eaadab25"} Nov 04 00:00:52.000358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.000282 6208 scope.go:122] "RemoveContainer" containerID="491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040" Nov 04 00:00:52.014942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.014919 6208 scope.go:122] "RemoveContainer" containerID="491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040" Nov 04 00:00:52.015259 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:00:52.015239 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040\": container with ID starting with 491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040 not found: ID does not exist" containerID="491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040" Nov 04 00:00:52.015312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.015268 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040"} err="failed to get container status \"491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040\": rpc error: code = NotFound desc = could not find container \"491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040\": container with ID starting with 491cd510d01ef2edc294938ff56092930d2c7329160b263de0b0518cd540d040 not found: ID does not exist" Nov 04 00:00:52.029347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.029317 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk"] Nov 04 00:00:52.034167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.034146 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-gmtdk"] Nov 04 00:00:52.394003 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.393928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:52.492132 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.492101 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:52.618061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:52.615849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:53.296023 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:53.295989 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" path="/var/lib/kubelet/pods/29e697bb-1d96-4ee6-aa78-78afce02e434/volumes" Nov 04 00:00:53.630560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:53.630476 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:53.899518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:53.898516 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:54.797398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:54.797363 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:55.023359 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:55.023274 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:55.459953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:55.459914 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:55.578119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:55.578077 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:56.369474 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:56.369420 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:56.677916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:56.677817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:56.786363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:56.786325 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:57.978399 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:57.978365 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:58.286789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:58.286698 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:59.370076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:59.370042 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:59.580951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:59.580917 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:00:59.878004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:00:59.877954 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:00.043923 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:00.043884 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:00.283592 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:00.283555 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:01.128073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:01.128019 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:02.444584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:02.444547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:02.718527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:02.717539 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n"] Nov 04 00:01:02.718527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:02.717820 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" containerID="cri-o://e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b" gracePeriod=30 Nov 04 00:01:03.441028 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:03.440997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:04.483847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:04.483815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:04.597323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:04.597287 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm"] Nov 04 00:01:04.597827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:04.597793 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" containerID="cri-o://9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669" gracePeriod=30 Nov 04 00:01:05.186006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:05.185971 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:05.343916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:05.343874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:05.681239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:05.681181 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:05.708261 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:01:05.708222 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:01:06.449955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:06.449917 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:06.598419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:06.598378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:06.715843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:06.715806 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:07.087903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:07.087811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:08.118665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.118608 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 04 00:01:08.129481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.129439 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:08.247012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.246980 6208 generic.go:356] "Generic (PLEG): container finished" podID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerID="e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b" exitCode=0 Nov 04 00:01:08.247012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.247017 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" event={"ID":"a3c58cce-5906-460b-81ca-0fe50b8881c1","Type":"ContainerDied","Data":"e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b"} Nov 04 00:01:08.247258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.247045 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" Nov 04 00:01:08.247258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.247074 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n" event={"ID":"a3c58cce-5906-460b-81ca-0fe50b8881c1","Type":"ContainerDied","Data":"e3c8cf67deeff8bec0c13b918ec197c2e51c95b57021ff461d8596482697f74b"} Nov 04 00:01:08.247258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.247107 6208 scope.go:122] "RemoveContainer" containerID="e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b" Nov 04 00:01:08.270222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.270184 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-credential-socket\") pod \"a3c58cce-5906-460b-81ca-0fe50b8881c1\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " Nov 04 00:01:08.270996 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.270884 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-credential-socket" pod "a3c58cce-5906-460b-81ca-0fe50b8881c1" (UID: "a3c58cce-5906-460b-81ca-0fe50b8881c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:08.271216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.271203 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c58cce-5906-460b-81ca-0fe50b8881c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c58cce-5906-460b-81ca-0fe50b8881c1-istiod-ca-cert\") pod \"a3c58cce-5906-460b-81ca-0fe50b8881c1\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " Nov 04 00:01:08.271616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.271597 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-socket\") pod \"a3c58cce-5906-460b-81ca-0fe50b8881c1\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " Nov 04 00:01:08.271944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.271932 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-podinfo\") pod \"a3c58cce-5906-460b-81ca-0fe50b8881c1\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " Nov 04 00:01:08.272458 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.271538 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c58cce-5906-460b-81ca-0fe50b8881c1-istiod-ca-cert" pod "a3c58cce-5906-460b-81ca-0fe50b8881c1" (UID: "a3c58cce-5906-460b-81ca-0fe50b8881c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:01:08.272458 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.271873 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-socket" pod "a3c58cce-5906-460b-81ca-0fe50b8881c1" (UID: "a3c58cce-5906-460b-81ca-0fe50b8881c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:08.272881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.272864 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-token\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-token\") pod \"a3c58cce-5906-460b-81ca-0fe50b8881c1\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " Nov 04 00:01:08.273378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.273359 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-certs\") pod \"a3c58cce-5906-460b-81ca-0fe50b8881c1\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " Nov 04 00:01:08.273841 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.273731 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-certs" pod "a3c58cce-5906-460b-81ca-0fe50b8881c1" (UID: "a3c58cce-5906-460b-81ca-0fe50b8881c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:08.273978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.273961 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-kube-api-access-mrb2x\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-kube-api-access-mrb2x\") pod \"a3c58cce-5906-460b-81ca-0fe50b8881c1\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " Nov 04 00:01:08.274515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.274498 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-data\") pod \"a3c58cce-5906-460b-81ca-0fe50b8881c1\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " Nov 04 00:01:08.275246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.274936 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-envoy\") pod \"a3c58cce-5906-460b-81ca-0fe50b8881c1\" (UID: \"a3c58cce-5906-460b-81ca-0fe50b8881c1\") " Nov 04 00:01:08.275450 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.274871 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-data" pod "a3c58cce-5906-460b-81ca-0fe50b8881c1" (UID: "a3c58cce-5906-460b-81ca-0fe50b8881c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:08.276497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.276476 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:08.278210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.278187 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:08.278438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.278378 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:08.279213 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.276910 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-podinfo" pod "a3c58cce-5906-460b-81ca-0fe50b8881c1" (UID: "a3c58cce-5906-460b-81ca-0fe50b8881c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:01:08.280194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.280174 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c58cce-5906-460b-81ca-0fe50b8881c1-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:08.280322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.280308 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:08.282892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.282358 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-token" pod "a3c58cce-5906-460b-81ca-0fe50b8881c1" (UID: "a3c58cce-5906-460b-81ca-0fe50b8881c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:01:08.282892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.282429 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-envoy" pod "a3c58cce-5906-460b-81ca-0fe50b8881c1" (UID: "a3c58cce-5906-460b-81ca-0fe50b8881c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:08.282892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.282648 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-kube-api-access-mrb2x" pod "a3c58cce-5906-460b-81ca-0fe50b8881c1" (UID: "a3c58cce-5906-460b-81ca-0fe50b8881c1"). InnerVolumeSpecName "kube-api-access-mrb2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:01:08.293996 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.293961 6208 scope.go:122] "RemoveContainer" containerID="e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b" Nov 04 00:01:08.294571 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:01:08.294538 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b\": container with ID starting with e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b not found: ID does not exist" containerID="e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b" Nov 04 00:01:08.294810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.294582 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b"} err="failed to get container status \"e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b\": rpc error: code = NotFound desc = could not find container \"e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b\": container with ID starting with e329ac5ae78e3727765e0c117a1d08844e517f51eb04cc37b893aa542562120b not found: ID does not exist" Nov 04 00:01:08.381518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.381483 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:08.381518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.381511 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrb2x\" (UniqueName: \"kubernetes.io/projected/a3c58cce-5906-460b-81ca-0fe50b8881c1-kube-api-access-mrb2x\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:08.381518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.381520 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:08.381745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.381530 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c58cce-5906-460b-81ca-0fe50b8881c1-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:08.582297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.582252 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n"] Nov 04 00:01:08.586787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:08.586729 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-dh64n"] Nov 04 00:01:09.033225 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:09.033174 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:09.295673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:09.295589 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" path="/var/lib/kubelet/pods/a3c58cce-5906-460b-81ca-0fe50b8881c1/volumes" Nov 04 00:01:09.935407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:09.935372 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 04 00:01:10.120986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.120943 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2385f4-0efd-4c2b-8291-327293f0db79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2385f4-0efd-4c2b-8291-327293f0db79-istiod-ca-cert\") pod \"7a2385f4-0efd-4c2b-8291-327293f0db79\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " Nov 04 00:01:10.121193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121152 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-kube-api-access-vnzdz\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-kube-api-access-vnzdz\") pod \"7a2385f4-0efd-4c2b-8291-327293f0db79\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " Nov 04 00:01:10.121246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121191 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-socket\") pod \"7a2385f4-0efd-4c2b-8291-327293f0db79\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " Nov 04 00:01:10.121246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121222 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-podinfo\") pod \"7a2385f4-0efd-4c2b-8291-327293f0db79\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " Nov 04 00:01:10.121690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121325 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2385f4-0efd-4c2b-8291-327293f0db79-istiod-ca-cert" pod "7a2385f4-0efd-4c2b-8291-327293f0db79" (UID: "7a2385f4-0efd-4c2b-8291-327293f0db79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:01:10.121690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121354 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-credential-socket\") pod \"7a2385f4-0efd-4c2b-8291-327293f0db79\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " Nov 04 00:01:10.121690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121475 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-socket" pod "7a2385f4-0efd-4c2b-8291-327293f0db79" (UID: "7a2385f4-0efd-4c2b-8291-327293f0db79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:10.121690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121577 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-data\") pod \"7a2385f4-0efd-4c2b-8291-327293f0db79\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " Nov 04 00:01:10.121690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121585 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-credential-socket" pod "7a2385f4-0efd-4c2b-8291-327293f0db79" (UID: "7a2385f4-0efd-4c2b-8291-327293f0db79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:10.121690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121646 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-envoy\") pod \"7a2385f4-0efd-4c2b-8291-327293f0db79\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " Nov 04 00:01:10.122082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121778 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-data" pod "7a2385f4-0efd-4c2b-8291-327293f0db79" (UID: "7a2385f4-0efd-4c2b-8291-327293f0db79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:10.122082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121804 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-token\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-token\") pod \"7a2385f4-0efd-4c2b-8291-327293f0db79\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " Nov 04 00:01:10.122082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.121855 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-certs\") pod \"7a2385f4-0efd-4c2b-8291-327293f0db79\" (UID: \"7a2385f4-0efd-4c2b-8291-327293f0db79\") " Nov 04 00:01:10.122256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.122192 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-certs" pod "7a2385f4-0efd-4c2b-8291-327293f0db79" (UID: "7a2385f4-0efd-4c2b-8291-327293f0db79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:10.122982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.122753 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:10.122982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.122781 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:10.122982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.122796 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:10.122982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.122826 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2385f4-0efd-4c2b-8291-327293f0db79-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:10.122982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.122840 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:10.123970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.123943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:10.124287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.124268 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-podinfo" pod "7a2385f4-0efd-4c2b-8291-327293f0db79" (UID: "7a2385f4-0efd-4c2b-8291-327293f0db79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:01:10.124655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.124631 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-token" pod "7a2385f4-0efd-4c2b-8291-327293f0db79" (UID: "7a2385f4-0efd-4c2b-8291-327293f0db79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:01:10.124742 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.124670 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-envoy" pod "7a2385f4-0efd-4c2b-8291-327293f0db79" (UID: "7a2385f4-0efd-4c2b-8291-327293f0db79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:10.124820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.124801 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-kube-api-access-vnzdz" pod "7a2385f4-0efd-4c2b-8291-327293f0db79" (UID: "7a2385f4-0efd-4c2b-8291-327293f0db79"). InnerVolumeSpecName "kube-api-access-vnzdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:01:10.224197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.224164 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:10.224197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.224192 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:10.224197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.224204 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnzdz\" (UniqueName: \"kubernetes.io/projected/7a2385f4-0efd-4c2b-8291-327293f0db79-kube-api-access-vnzdz\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:10.224447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.224214 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2385f4-0efd-4c2b-8291-327293f0db79-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:10.265609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.265577 6208 generic.go:356] "Generic (PLEG): container finished" podID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerID="9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669" exitCode=0 Nov 04 00:01:10.265760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.265629 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" Nov 04 00:01:10.265760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.265664 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" event={"ID":"7a2385f4-0efd-4c2b-8291-327293f0db79","Type":"ContainerDied","Data":"9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669"} Nov 04 00:01:10.265760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.265723 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm" event={"ID":"7a2385f4-0efd-4c2b-8291-327293f0db79","Type":"ContainerDied","Data":"c0c257c45959f9a880139cc8934963728590e23a0a2b5b9f8a36c813cd143b25"} Nov 04 00:01:10.265760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.265753 6208 scope.go:122] "RemoveContainer" containerID="9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669" Nov 04 00:01:10.287675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.287653 6208 scope.go:122] "RemoveContainer" containerID="9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669" Nov 04 00:01:10.287985 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:01:10.287950 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669\": container with ID starting with 9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669 not found: ID does not exist" containerID="9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669" Nov 04 00:01:10.288128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.287995 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669"} err="failed to get container status \"9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669\": rpc error: code = NotFound desc = could not find container \"9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669\": container with ID starting with 9facb4ac01c2e700fb09d6f206270b9d01519daa53c9653916fe2b48d32ac669 not found: ID does not exist" Nov 04 00:01:10.297500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.297475 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm"] Nov 04 00:01:10.301738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.301717 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-c5tqm"] Nov 04 00:01:10.768310 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:10.768276 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:11.295250 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.295214 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" path="/var/lib/kubelet/pods/7a2385f4-0efd-4c2b-8291-327293f0db79/volumes" Nov 04 00:01:11.340389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.340352 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:11.453865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.453826 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:11.775024 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.774982 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:11.826954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.826920 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x"] Nov 04 00:01:11.827980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.827885 6208 cpu_manager.go:414] "containerMap: removing container" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerName="cleanup" Nov 04 00:01:11.827980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.827912 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f57d3c-6e1a-4cb2-82d6-969494d04963" containerName="cleanup" Nov 04 00:01:11.827980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.827928 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" Nov 04 00:01:11.827980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.827936 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" Nov 04 00:01:11.827980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.827958 6208 cpu_manager.go:414] "containerMap: removing container" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" Nov 04 00:01:11.827980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.827965 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" Nov 04 00:01:11.828273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.828007 6208 cpu_manager.go:414] "containerMap: removing container" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" Nov 04 00:01:11.828273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.828015 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" Nov 04 00:01:11.828273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.828053 6208 cpu_manager.go:414] "containerMap: removing container" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerName="cleanup" Nov 04 00:01:11.828273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.828060 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerName="cleanup" Nov 04 00:01:11.828273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.828241 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2385f4-0efd-4c2b-8291-327293f0db79" containerName="istio-proxy" Nov 04 00:01:11.828273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.828258 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e697bb-1d96-4ee6-aa78-78afce02e434" containerName="istio-proxy" Nov 04 00:01:11.828273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.828270 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26dbe69-425e-4f9e-8758-4d86d7d2aeb3" containerName="cleanup" Nov 04 00:01:11.828508 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.828287 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c58cce-5906-460b-81ca-0fe50b8881c1" containerName="istio-proxy" Nov 04 00:01:11.842748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.842722 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:11.848112 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.848083 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-18f5c0-openshift-default-dockercfg-7c66n\"" Nov 04 00:01:11.876513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.876022 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x"] Nov 04 00:01:11.947219 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.947189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:11.957522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.957489 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2mr\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-kube-api-access-cj2mr\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:11.957649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.957635 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-token\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:11.957696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.957675 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-data\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:11.957739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.957698 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-certs\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:11.957841 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.957819 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad550b3-aad8-40e6-80b2-98d821384e5e-istiod-ca-cert\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:11.957897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.957881 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-podinfo\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:11.957941 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.957921 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-socket\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:11.958008 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.957993 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-credential-socket\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:11.958101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:11.958083 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-envoy\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.060243 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad550b3-aad8-40e6-80b2-98d821384e5e-istiod-ca-cert\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.060330 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-podinfo\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.060383 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-socket\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.060440 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-credential-socket\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.060503 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-envoy\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.060557 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2mr\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-kube-api-access-cj2mr\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.060706 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-token\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.060754 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-data\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.060787 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-certs\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.061282 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-certs\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.062308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.061919 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-socket\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.067299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.062658 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-credential-socket\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.067299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.063430 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-data\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.067299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.064114 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad550b3-aad8-40e6-80b2-98d821384e5e-istiod-ca-cert\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.111112 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.091232 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-envoy\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.111112 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.093352 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-podinfo\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.115258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.115226 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-token\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.116097 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.116055 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2mr\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-kube-api-access-cj2mr\") pod \"e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.178102 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.175295 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:12.191053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.190458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:12.553334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.553305 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x"] Nov 04 00:01:12.555103 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:01:12.555070 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad550b3_aad8_40e6_80b2_98d821384e5e.slice/crio-c5c2a4d2ba777701bca6a024d5ee59cbd2d1e10177ab9ad070fe231ab3f1fd68 WatchSource:0}: Error finding container c5c2a4d2ba777701bca6a024d5ee59cbd2d1e10177ab9ad070fe231ab3f1fd68: Status 404 returned error can't find the container with id c5c2a4d2ba777701bca6a024d5ee59cbd2d1e10177ab9ad070fe231ab3f1fd68 Nov 04 00:01:12.558160 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.558112 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:12.558261 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.558194 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:12.558261 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.558229 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:12.718408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.660702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:12.780054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.779392 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:12.930100 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:01:12.930069 6208 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418b6385_c834_4532_94d5_e65086de1352.slice/crio-6f2e4ee658d9b0c43dce210c1c2bc7954ce1b72b510cb3bb236892142a49605f.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@17-32050-0.service\": RecentStats: unable to find data in memory cache]" Nov 04 00:01:12.963044 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:12.962971 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:13.293868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.293832 6208 generic.go:356] "Generic (PLEG): container finished" podID="418b6385-c834-4532-94d5-e65086de1352" containerID="6f2e4ee658d9b0c43dce210c1c2bc7954ce1b72b510cb3bb236892142a49605f" exitCode=0 Nov 04 00:01:13.297062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.297017 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" event={"ID":"4ad550b3-aad8-40e6-80b2-98d821384e5e","Type":"ContainerStarted","Data":"4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302"} Nov 04 00:01:13.297189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.297084 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" event={"ID":"4ad550b3-aad8-40e6-80b2-98d821384e5e","Type":"ContainerStarted","Data":"c5c2a4d2ba777701bca6a024d5ee59cbd2d1e10177ab9ad070fe231ab3f1fd68"} Nov 04 00:01:13.297189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.297111 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" event={"ID":"418b6385-c834-4532-94d5-e65086de1352","Type":"ContainerDied","Data":"6f2e4ee658d9b0c43dce210c1c2bc7954ce1b72b510cb3bb236892142a49605f"} Nov 04 00:01:13.297555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.297540 6208 scope.go:122] "RemoveContainer" containerID="6f2e4ee658d9b0c43dce210c1c2bc7954ce1b72b510cb3bb236892142a49605f" Nov 04 00:01:13.299218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.299185 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:13.299348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.299270 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:13.299348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.299322 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:13.327449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.327380 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" podStartSLOduration=2.327365083 podStartE2EDuration="2.327365083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:13.32681246 +0000 UTC m=+1892.149408623" watchObservedRunningTime="2026-11-04 00:01:13.327365083 +0000 UTC m=+1892.149961245" Nov 04 00:01:13.981052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.979557 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc"] Nov 04 00:01:13.981052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:13.979796 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" podUID="0d80a176-9ae6-4d75-8088-26091310280b" containerName="discovery" containerID="cri-o://94db4dcc6ee049bb3e0942bb3cab069e52dae3d095925aa44e845fbc8ffa23c0" gracePeriod=30 Nov 04 00:01:14.180056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.175813 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:14.181915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.181885 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:14.185455 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.185404 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:14.480175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.480134 6208 generic.go:356] "Generic (PLEG): container finished" podID="0d80a176-9ae6-4d75-8088-26091310280b" containerID="94db4dcc6ee049bb3e0942bb3cab069e52dae3d095925aa44e845fbc8ffa23c0" exitCode=0 Nov 04 00:01:14.480376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.480302 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" event={"ID":"0d80a176-9ae6-4d75-8088-26091310280b","Type":"ContainerDied","Data":"94db4dcc6ee049bb3e0942bb3cab069e52dae3d095925aa44e845fbc8ffa23c0"} Nov 04 00:01:14.484405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.484318 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" event={"ID":"418b6385-c834-4532-94d5-e65086de1352","Type":"ContainerStarted","Data":"e6a8c63690629c9b2cc658c27bb8353142a0921805a8a2ffa301efbc0e769560"} Nov 04 00:01:14.484405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.484388 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:14.487101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.487073 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:01:14.529359 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.529334 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 04 00:01:14.549157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.549129 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-dns-cert\") pod \"0d80a176-9ae6-4d75-8088-26091310280b\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " Nov 04 00:01:14.549274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.549172 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d80a176-9ae6-4d75-8088-26091310280b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d80a176-9ae6-4d75-8088-26091310280b-local-certs\") pod \"0d80a176-9ae6-4d75-8088-26091310280b\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " Nov 04 00:01:14.549274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.549245 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-ca-configmap\") pod \"0d80a176-9ae6-4d75-8088-26091310280b\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " Nov 04 00:01:14.549358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.549284 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-cacerts\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-cacerts\") pod \"0d80a176-9ae6-4d75-8088-26091310280b\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " Nov 04 00:01:14.549410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.549394 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-istio-token\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-istio-token\") pod \"0d80a176-9ae6-4d75-8088-26091310280b\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " Nov 04 00:01:14.549516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.549500 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-kubeconfig\") pod \"0d80a176-9ae6-4d75-8088-26091310280b\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " Nov 04 00:01:14.549651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.549594 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-kube-api-access-j9wbp\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-kube-api-access-j9wbp\") pod \"0d80a176-9ae6-4d75-8088-26091310280b\" (UID: \"0d80a176-9ae6-4d75-8088-26091310280b\") " Nov 04 00:01:14.549727 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.549649 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-ca-configmap" pod "0d80a176-9ae6-4d75-8088-26091310280b" (UID: "0d80a176-9ae6-4d75-8088-26091310280b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:01:14.550537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.550507 6208 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-ca-configmap\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:14.551842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.551813 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-istio-token" pod "0d80a176-9ae6-4d75-8088-26091310280b" (UID: "0d80a176-9ae6-4d75-8088-26091310280b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:01:14.551961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.551930 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-dns-cert" pod "0d80a176-9ae6-4d75-8088-26091310280b" (UID: "0d80a176-9ae6-4d75-8088-26091310280b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:01:14.552065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.551976 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-kubeconfig" pod "0d80a176-9ae6-4d75-8088-26091310280b" (UID: "0d80a176-9ae6-4d75-8088-26091310280b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:01:14.552139 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.552117 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-kube-api-access-j9wbp" pod "0d80a176-9ae6-4d75-8088-26091310280b" (UID: "0d80a176-9ae6-4d75-8088-26091310280b"). InnerVolumeSpecName "kube-api-access-j9wbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:01:14.552222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.552205 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-cacerts" pod "0d80a176-9ae6-4d75-8088-26091310280b" (UID: "0d80a176-9ae6-4d75-8088-26091310280b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:01:14.552547 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.552524 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d80a176-9ae6-4d75-8088-26091310280b-local-certs" pod "0d80a176-9ae6-4d75-8088-26091310280b" (UID: "0d80a176-9ae6-4d75-8088-26091310280b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:01:14.651836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.651801 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:14.651836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.651834 6208 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-kubeconfig\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:14.652106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.651848 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wbp\" (UniqueName: \"kubernetes.io/projected/0d80a176-9ae6-4d75-8088-26091310280b-kube-api-access-j9wbp\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:14.652106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.651864 6208 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-istio-csr-dns-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:14.652106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.651876 6208 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d80a176-9ae6-4d75-8088-26091310280b-local-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:14.652106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.651891 6208 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d80a176-9ae6-4d75-8088-26091310280b-cacerts\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:01:14.692118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.692076 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:01:14.692467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.692440 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 00:01:14.692572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:14.692511 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 00:01:15.373056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:15.369525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:15.497516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:15.497487 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" Nov 04 00:01:15.497712 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:15.497514 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc" event={"ID":"0d80a176-9ae6-4d75-8088-26091310280b","Type":"ContainerDied","Data":"d82363982fea48b1dac5888cdc2036439726c746c7dd85354913c32b852f1231"} Nov 04 00:01:15.497712 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:15.497576 6208 scope.go:122] "RemoveContainer" containerID="94db4dcc6ee049bb3e0942bb3cab069e52dae3d095925aa44e845fbc8ffa23c0" Nov 04 00:01:15.526952 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:15.526915 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc"] Nov 04 00:01:15.530807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:15.530782 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dhdlc"] Nov 04 00:01:15.692120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:15.692004 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 00:01:15.692120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:15.692073 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 00:01:16.383684 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:16.383646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:16.691346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:16.691254 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 00:01:16.691346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:16.691316 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 00:01:17.295551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.295517 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d80a176-9ae6-4d75-8088-26091310280b" path="/var/lib/kubelet/pods/0d80a176-9ae6-4d75-8088-26091310280b/volumes" Nov 04 00:01:17.385057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.378412 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c"] Nov 04 00:01:17.385057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.382510 6208 cpu_manager.go:414] "containerMap: removing container" podUID="0d80a176-9ae6-4d75-8088-26091310280b" containerName="discovery" Nov 04 00:01:17.385057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.382535 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d80a176-9ae6-4d75-8088-26091310280b" containerName="discovery" Nov 04 00:01:17.385057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.382700 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d80a176-9ae6-4d75-8088-26091310280b" containerName="discovery" Nov 04 00:01:17.395791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.395534 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.407107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.407077 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-3a002b-serving-cert\"" Nov 04 00:01:17.407922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.407894 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c"] Nov 04 00:01:17.483207 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.477154 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:17.543729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.542676 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.543729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.542756 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79986d29-e237-4bbc-bb59-9687d182ca4c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.543729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.542807 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27kp\" (UniqueName: \"kubernetes.io/projected/79986d29-e237-4bbc-bb59-9687d182ca4c-kube-api-access-j27kp\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.543729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.542840 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-metrics-certs\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.595411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.595296 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:17.644027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.643993 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.644258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.644061 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79986d29-e237-4bbc-bb59-9687d182ca4c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.644258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.644104 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27kp\" (UniqueName: \"kubernetes.io/projected/79986d29-e237-4bbc-bb59-9687d182ca4c-kube-api-access-j27kp\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.644258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.644127 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-metrics-certs\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.644874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.644844 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79986d29-e237-4bbc-bb59-9687d182ca4c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.646593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.646567 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-metrics-certs\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.646716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.646649 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.686741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.686695 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27kp\" (UniqueName: \"kubernetes.io/projected/79986d29-e237-4bbc-bb59-9687d182ca4c-kube-api-access-j27kp\") pod \"maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.692838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.691739 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 00:01:17.692838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.691793 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 00:01:17.711444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.711414 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7"] Nov 04 00:01:17.720052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.717743 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.726085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.724809 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:17.747226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.747083 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:17.752496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.752227 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7"] Nov 04 00:01:17.849823 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.849791 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.849954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.849908 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.850012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.849972 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9v86\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-kube-api-access-j9v86\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.850116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.850104 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.850212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.850192 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.850279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.850233 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.850338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.850290 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.850394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.850345 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.850447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.850397 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952211 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952307 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952373 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952408 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9v86\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-kube-api-access-j9v86\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952445 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952488 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952507 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952542 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952544 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-credential-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.952594 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.971193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.953190 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-socket\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.979303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.978556 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.979303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.978894 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-data\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.979303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.978940 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-certs\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.982601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.982555 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-envoy\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.983222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.983167 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-token\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.984100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.983918 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-podinfo\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:17.984829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:17.984803 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9v86\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-kube-api-access-j9v86\") pod \"e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:18.042527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.042488 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:18.092150 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.091643 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c"] Nov 04 00:01:18.113818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.113782 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:01:18.442614 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.442588 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7"] Nov 04 00:01:18.443354 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:01:18.443324 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7eef87_24c5_4fe0_a7dd_1f766c7a906f.slice/crio-63bef449e2a0325115483f85eba5620dbde38772f0a4ef36f89e789c6d1831eb WatchSource:0}: Error finding container 63bef449e2a0325115483f85eba5620dbde38772f0a4ef36f89e789c6d1831eb: Status 404 returned error can't find the container with id 63bef449e2a0325115483f85eba5620dbde38772f0a4ef36f89e789c6d1831eb Nov 04 00:01:18.446172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.446139 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:18.446254 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.446205 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:18.446254 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.446236 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:18.579064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.574937 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" event={"ID":"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f","Type":"ContainerStarted","Data":"943655b5d5a40214148bac172f5aab6f7a4258ef2a9c574a617bec000909ce52"} Nov 04 00:01:18.579064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.575003 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" event={"ID":"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f","Type":"ContainerStarted","Data":"63bef449e2a0325115483f85eba5620dbde38772f0a4ef36f89e789c6d1831eb"} Nov 04 00:01:18.586086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.584637 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" event={"ID":"79986d29-e237-4bbc-bb59-9687d182ca4c","Type":"ContainerStarted","Data":"c7e4e0d0fc5f76cd600730faf42feef1cd74e6957c54d453fd6e65b5d7ab340c"} Nov 04 00:01:18.606333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.606255 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" podStartSLOduration=1.6062420899999998 podStartE2EDuration="1.60624209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:18.603389943 +0000 UTC m=+1897.425986107" watchObservedRunningTime="2026-11-04 00:01:18.60624209 +0000 UTC m=+1897.428838251" Nov 04 00:01:18.691530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.691358 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 04 00:01:18.691721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.691440 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 04 00:01:18.738397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:18.738249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:19.046174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.043925 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:19.046174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.044521 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 00:01:19.046174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.044575 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 00:01:19.595987 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.595951 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" event={"ID":"79986d29-e237-4bbc-bb59-9687d182ca4c","Type":"ContainerStarted","Data":"3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be"} Nov 04 00:01:19.596389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.596271 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:19.627760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.627653 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" podStartSLOduration=2.139856456 podStartE2EDuration="2.627634203s" totalImagesPullingTime="487.777747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:19.625678683 +0000 UTC m=+1898.448274866" watchObservedRunningTime="2026-11-04 00:01:19.627634203 +0000 UTC m=+1898.450230366" Nov 04 00:01:19.838308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.838222 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:19.916804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.916683 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:01:19.918008 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.917984 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:01:19.918999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:19.918981 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:01:20.043185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:20.043154 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 00:01:20.043337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:20.043206 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 00:01:20.885835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:20.885796 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:21.043391 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:21.043341 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 00:01:21.043588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:21.043412 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 00:01:21.889087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:21.889049 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:22.043836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:22.043789 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 00:01:22.044018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:22.043859 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 00:01:22.980065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:22.979980 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:23.042928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:23.042887 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 00:01:23.043126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:23.042970 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 00:01:24.043935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:24.043899 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 04 00:01:24.044383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:24.043961 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 04 00:01:24.175521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:24.175481 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:25.191565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:25.191534 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:25.192529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:25.192505 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:25.193577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:25.193553 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:01:25.524167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:25.524134 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:25.609403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:25.609367 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:01:25.683940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:25.683900 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:25.807405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:25.807305 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:27.038879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.038835 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:27.191075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.191008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:27.223962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.223931 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z"] Nov 04 00:01:27.229609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.229583 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:27.239741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.239709 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-18f5c0\"" Nov 04 00:01:27.239989 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.239969 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-18f5c0-dockercfg-mfhmh\"" Nov 04 00:01:27.240242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.240223 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z"] Nov 04 00:01:27.273654 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.273624 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4c2c\" (UniqueName: \"kubernetes.io/projected/a2177c20-6e16-48e7-948a-78fa2ce0fc66-kube-api-access-h4c2c\") pod \"payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z\" (UID: \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:27.273922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.273898 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2177c20-6e16-48e7-948a-78fa2ce0fc66-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z\" (UID: \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:27.343226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.343147 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:27.375819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.375785 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2177c20-6e16-48e7-948a-78fa2ce0fc66-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z\" (UID: \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:27.375985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.375915 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4c2c\" (UniqueName: \"kubernetes.io/projected/a2177c20-6e16-48e7-948a-78fa2ce0fc66-kube-api-access-h4c2c\") pod \"payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z\" (UID: \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:27.376555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.376534 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2177c20-6e16-48e7-948a-78fa2ce0fc66-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z\" (UID: \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:27.388938 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.388911 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4c2c\" (UniqueName: \"kubernetes.io/projected/a2177c20-6e16-48e7-948a-78fa2ce0fc66-kube-api-access-h4c2c\") pod \"payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z\" (UID: \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:27.499100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.493099 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:27.547568 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.547295 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:27.877509 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:01:27.877479 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2177c20_6e16_48e7_948a_78fa2ce0fc66.slice/crio-9b03db8c1964a1ae22a20f85b6f1abc727f10bf45f241a28f28b881c0f24411f WatchSource:0}: Error finding container 9b03db8c1964a1ae22a20f85b6f1abc727f10bf45f241a28f28b881c0f24411f: Status 404 returned error can't find the container with id 9b03db8c1964a1ae22a20f85b6f1abc727f10bf45f241a28f28b881c0f24411f Nov 04 00:01:27.878103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:27.878003 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z"] Nov 04 00:01:28.660183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:28.660133 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:28.702590 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:28.702547 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" event={"ID":"a2177c20-6e16-48e7-948a-78fa2ce0fc66","Type":"ContainerStarted","Data":"05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f"} Nov 04 00:01:28.702795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:28.702650 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" event={"ID":"a2177c20-6e16-48e7-948a-78fa2ce0fc66","Type":"ContainerStarted","Data":"9b03db8c1964a1ae22a20f85b6f1abc727f10bf45f241a28f28b881c0f24411f"} Nov 04 00:01:28.702889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:28.702827 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:28.728447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:28.728378 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" podStartSLOduration=1.7283589560000001 podStartE2EDuration="1.728358956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:28.723815672 +0000 UTC m=+1907.546411846" watchObservedRunningTime="2026-11-04 00:01:28.728358956 +0000 UTC m=+1907.550955119" Nov 04 00:01:29.618675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:29.618635 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:30.491903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:30.491867 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:31.342568 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:31.342525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:32.455193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:32.455160 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:33.258441 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:33.258406 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:33.546237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:33.546135 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:34.086059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:34.085528 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:35.188245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.188213 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:35.415534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.415498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:35.672747 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.672713 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr"] Nov 04 00:01:35.678313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.678282 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.686924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.686892 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8895cb-openshift-default-dockercfg-lxj7b\"" Nov 04 00:01:35.694363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.694332 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr"] Nov 04 00:01:35.748197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.748161 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:35.794581 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:01:35.794544 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:01:35.880831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.880795 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-data\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.881144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.881118 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-istio-token\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.881285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.881271 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-envoy\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.881448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.881432 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-credential-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.881556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.881544 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-certs\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.881655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.881644 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b9470d-1121-4176-be18-bd5348ae17db-istio-podinfo\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.881783 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.881770 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.881915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.881901 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b9470d-1121-4176-be18-bd5348ae17db-istiod-ca-cert\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.882151 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.882135 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jx4s\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-kube-api-access-2jx4s\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.948113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.943946 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:35.991059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.987611 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-credential-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.991059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.987667 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-certs\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.991059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.987710 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b9470d-1121-4176-be18-bd5348ae17db-istio-podinfo\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.991059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.987767 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.991059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.987830 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b9470d-1121-4176-be18-bd5348ae17db-istiod-ca-cert\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.991059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.987955 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jx4s\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-kube-api-access-2jx4s\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.991059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.988051 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-data\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.991059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.988109 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-istio-token\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:35.991059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.988160 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-envoy\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.002052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.996614 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.002052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.996876 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-credential-socket\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.002052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.997146 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-certs\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.002052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.998567 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b9470d-1121-4176-be18-bd5348ae17db-istiod-ca-cert\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.002052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:35.998792 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-data\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.055091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.051603 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-envoy\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.084531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.062641 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-istio-token\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.084531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.066440 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b9470d-1121-4176-be18-bd5348ae17db-istio-podinfo\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.105058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.095165 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jx4s\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-kube-api-access-2jx4s\") pod \"e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.323346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.323312 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:36.631230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.630596 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:36.727486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.726898 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr"] Nov 04 00:01:36.731096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.728223 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:36.731096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.728327 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:36.731096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.728368 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:36.819347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:36.819314 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" event={"ID":"77b9470d-1121-4176-be18-bd5348ae17db","Type":"ContainerStarted","Data":"8a4825d33d1f7f56ad5fd30f8e085428c04e657140572b06e1f99a918330b42e"} Nov 04 00:01:37.339587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:37.339555 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:37.638196 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:37.638104 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:37.828802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:37.828762 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" event={"ID":"77b9470d-1121-4176-be18-bd5348ae17db","Type":"ContainerStarted","Data":"72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7"} Nov 04 00:01:37.881980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:37.881682 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podStartSLOduration=2.8816618 podStartE2EDuration="2.8816618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:37.874414058 +0000 UTC m=+1916.697010221" watchObservedRunningTime="2026-11-04 00:01:37.8816618 +0000 UTC m=+1916.704257964" Nov 04 00:01:38.102112 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:38.102073 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:38.323450 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:38.323414 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:38.329640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:38.329612 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:38.448740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:38.448652 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:38.842971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:38.842935 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:38.843962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:38.843940 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:01:39.379461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:39.379424 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:39.716950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:39.716924 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:01:39.981630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:39.981517 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:40.543186 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:40.543132 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:41.344393 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:41.344364 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:41.452244 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:41.452206 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:42.333989 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:42.333951 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:42.720406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:42.720373 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:43.096152 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.096061 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55"] Nov 04 00:01:43.101753 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.101725 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.112086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.110574 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-1b5fca-openshift-default-dockercfg-jndcp\"" Nov 04 00:01:43.120901 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.120547 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55"] Nov 04 00:01:43.145296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.143723 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:43.337080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.325408 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-envoy\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.337080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.325524 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-certs\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.337080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.325567 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ljzq\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-kube-api-access-2ljzq\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.337080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.325663 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-data\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.337080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.325720 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-token\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.337080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.325779 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5979d19-2211-458a-9734-0a9a8ddfd617-istiod-ca-cert\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.337080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.325903 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-podinfo\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.337080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.326143 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-socket\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.337080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.326183 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-credential-socket\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.422535 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.428088 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-socket\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.428151 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-credential-socket\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.428216 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-envoy\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.428279 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-certs\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.428308 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ljzq\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-kube-api-access-2ljzq\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.428385 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-data\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.428429 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-token\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.428481 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5979d19-2211-458a-9734-0a9a8ddfd617-istiod-ca-cert\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.428579 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-podinfo\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.429204 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-socket\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.429470 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-credential-socket\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.430444 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5979d19-2211-458a-9734-0a9a8ddfd617-istiod-ca-cert\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.432977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.432763 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-data\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.433055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.432992 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-certs\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.447774 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.447696 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ljzq\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-kube-api-access-2ljzq\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.449699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.449669 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-token\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.450922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.450893 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-envoy\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.451083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.451004 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-podinfo\") pod \"e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:43.572683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.572647 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:43.731360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:43.731331 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:44.165339 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:44.165304 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55"] Nov 04 00:01:44.165820 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:01:44.165790 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5979d19_2211_458a_9734_0a9a8ddfd617.slice/crio-34d7906197471247e08acabe00a532919d94509e20d16ec966967ac8b2a9fb4e WatchSource:0}: Error finding container 34d7906197471247e08acabe00a532919d94509e20d16ec966967ac8b2a9fb4e: Status 404 returned error can't find the container with id 34d7906197471247e08acabe00a532919d94509e20d16ec966967ac8b2a9fb4e Nov 04 00:01:44.169115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:44.169069 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:44.169203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:44.169166 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:44.169240 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:44.169212 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:44.232759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:44.232725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:44.915752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:44.915717 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" event={"ID":"f5979d19-2211-458a-9734-0a9a8ddfd617","Type":"ContainerStarted","Data":"eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df"} Nov 04 00:01:44.916267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:44.915772 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" event={"ID":"f5979d19-2211-458a-9734-0a9a8ddfd617","Type":"ContainerStarted","Data":"34d7906197471247e08acabe00a532919d94509e20d16ec966967ac8b2a9fb4e"} Nov 04 00:01:44.950533 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:44.949825 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podStartSLOduration=1.949804641 podStartE2EDuration="1.949804641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:44.939930531 +0000 UTC m=+1923.762526693" watchObservedRunningTime="2026-11-04 00:01:44.949804641 +0000 UTC m=+1923.772400804" Nov 04 00:01:45.098488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:45.098449 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:45.731548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:45.731496 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:45.740405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:45.740374 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:45.925382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:45.925349 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:45.926422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:45.926399 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:01:46.054962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:46.054879 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:46.166797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:46.166758 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:47.424807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:47.423782 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:47.556187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:47.556147 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:47.671542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:47.671505 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:48.721645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:48.721613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:48.875321 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:48.875280 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:49.006619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:49.006498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:50.247344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:50.247312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:50.379006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:50.378970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:50.486881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:50.486846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:50.852593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:50.852558 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:51.343983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:51.343950 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:51.609147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:51.608989 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:51.719274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:51.719240 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:52.001205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.001166 6208 generic.go:356] "Generic (PLEG): container finished" podID="676b3179-ba28-4ce1-baa2-d3557794a632" containerID="12fa0f85e44d5ef6bc2c8288efd2d317a883259b250a36bbca3eb2a42653454d" exitCode=0 Nov 04 00:01:52.001407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.001226 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" event={"ID":"676b3179-ba28-4ce1-baa2-d3557794a632","Type":"ContainerDied","Data":"12fa0f85e44d5ef6bc2c8288efd2d317a883259b250a36bbca3eb2a42653454d"} Nov 04 00:01:52.001751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.001725 6208 scope.go:122] "RemoveContainer" containerID="12fa0f85e44d5ef6bc2c8288efd2d317a883259b250a36bbca3eb2a42653454d" Nov 04 00:01:52.005978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.005937 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:52.006263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.006237 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:52.006430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.006407 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:01:52.010093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.010071 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:52.153908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.153873 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:52.286737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.286651 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:52.543437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.543328 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:52.790114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.790075 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:52.915735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:52.915636 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:53.010098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:53.010064 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" event={"ID":"676b3179-ba28-4ce1-baa2-d3557794a632","Type":"ContainerStarted","Data":"20b58e7c93c99efba0c738e8ee7cf778c0cfec59bf5e867612186d439f33ca21"} Nov 04 00:01:53.315967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:53.315864 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:53.383574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:53.383391 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:01:53.384783 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:53.384749 6208 patch_prober.go:29] interesting pod/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 00:01:53.384916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:53.384814 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 00:01:53.437647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:53.437608 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:53.893833 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:53.893802 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:54.388192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.388087 6208 patch_prober.go:29] interesting pod/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 00:01:54.388192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.388168 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 00:01:54.549968 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.549931 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:54.590762 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.590721 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5"] Nov 04 00:01:54.594210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.594191 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.602412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.602387 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"routing-overlay\"" Nov 04 00:01:54.602504 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.602478 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-e2e--5d63ff17-dockercfg-lrbbg\"" Nov 04 00:01:54.602570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.602517 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-acf8cd35b468\"" Nov 04 00:01:54.602912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.602890 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"openshift-service-ca.crt\"" Nov 04 00:01:54.603009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.602973 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2697a0\"" Nov 04 00:01:54.603119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.603103 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"kube-root-ca.crt\"" Nov 04 00:01:54.607137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.607114 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5"] Nov 04 00:01:54.686075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.685974 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.686240 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.686208 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.686315 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.686299 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.686377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.686352 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gj9q\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-kube-api-access-2gj9q\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.798055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.788112 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.798055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.788236 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.798055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.788277 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gj9q\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-kube-api-access-2gj9q\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.798055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.788418 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.798055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.789701 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.798055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.790500 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.847057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.846743 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gj9q\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-kube-api-access-2gj9q\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.852916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.851642 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:54.913124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:54.912351 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:55.049195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:55.049008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:55.383795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:55.383700 6208 patch_prober.go:29] interesting pod/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 00:01:55.383795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:55.383773 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 00:01:55.438065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:55.435826 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5"] Nov 04 00:01:55.446477 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:01:55.446442 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c3f135_3c06_451e_b048_ecc1fc1cf2e4.slice/crio-279f768e49fcc4f036e001e8edf480686a41e2eaff821fea4b135086e1168188 WatchSource:0}: Error finding container 279f768e49fcc4f036e001e8edf480686a41e2eaff821fea4b135086e1168188: Status 404 returned error can't find the container with id 279f768e49fcc4f036e001e8edf480686a41e2eaff821fea4b135086e1168188 Nov 04 00:01:55.486668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:55.486627 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:55.917247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:55.917145 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:01:55.997057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:55.993855 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:56.097122 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:56.096822 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:56.181836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:56.181740 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" event={"ID":"77c3f135-3c06-451e-b048-ecc1fc1cf2e4","Type":"ContainerStarted","Data":"6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac"} Nov 04 00:01:56.181836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:56.181810 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" event={"ID":"77c3f135-3c06-451e-b048-ecc1fc1cf2e4","Type":"ContainerStarted","Data":"279f768e49fcc4f036e001e8edf480686a41e2eaff821fea4b135086e1168188"} Nov 04 00:01:56.182085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:56.181840 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:01:56.209662 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:56.209603 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" podStartSLOduration=2.209587268 podStartE2EDuration="2.209587268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:01:56.207645623 +0000 UTC m=+1935.030241800" watchObservedRunningTime="2026-11-04 00:01:56.209587268 +0000 UTC m=+1935.032183430" Nov 04 00:01:56.384522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:56.384485 6208 patch_prober.go:29] interesting pod/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 00:01:56.384744 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:56.384554 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 00:01:56.730499 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:56.730467 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:57.384374 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:57.384335 6208 patch_prober.go:29] interesting pod/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 00:01:57.384839 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:57.384399 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 00:01:57.550919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:57.550849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:58.251927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:58.251881 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:58.383605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:58.383539 6208 patch_prober.go:29] interesting pod/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 04 00:01:58.385052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:58.383730 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 04 00:01:59.142478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:59.142443 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:01:59.390136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:59.390089 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:01:59.392681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:59.392615 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:01:59.967620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:01:59.967578 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:00.840057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:00.839994 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:01.038661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:01.038612 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:01.689848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:01.689760 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:01.791992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:01.791928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:02.549203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:02.549172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:03.493653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:03.493585 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:04.361738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:04.361703 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:05.151028 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:05.150980 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:05.707507 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:02:05.707469 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:02:05.932427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:05.932393 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:06.718697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:06.718663 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:06.848175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:06.848127 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:07.192810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:07.192732 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:02:08.109424 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.109383 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8"] Nov 04 00:02:08.113862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.113836 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.118328 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.118301 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-1b5fca-serving-cert\"" Nov 04 00:02:08.132471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.132440 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8"] Nov 04 00:02:08.155924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.155885 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.156117 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.155999 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64445549-ad95-4060-9522-ab6b550cecbf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.156191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.156110 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-metrics-certs\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.156247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.156207 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppndz\" (UniqueName: \"kubernetes.io/projected/64445549-ad95-4060-9522-ab6b550cecbf-kube-api-access-ppndz\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.228619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.228581 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:08.258363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.258330 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.258567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.258457 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64445549-ad95-4060-9522-ab6b550cecbf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.258567 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:02:08.258512 6208 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-serving-cert: secret "maas-api-e2e-extmod-b-1b5fca-serving-cert" not found Nov 04 00:02:08.258694 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:02:08.258647 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls podName:64445549-ad95-4060-9522-ab6b550cecbf nodeName:}" failed. No retries permitted until 2026-11-04 00:02:08.758622477 +0000 UTC m=+1947.581218631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls") pod "maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" (UID: "64445549-ad95-4060-9522-ab6b550cecbf") : secret "maas-api-e2e-extmod-b-1b5fca-serving-cert" not found Nov 04 00:02:08.258694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.258538 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-metrics-certs\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.258824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.258803 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppndz\" (UniqueName: \"kubernetes.io/projected/64445549-ad95-4060-9522-ab6b550cecbf-kube-api-access-ppndz\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.259227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.259203 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64445549-ad95-4060-9522-ab6b550cecbf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.261215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.261187 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-metrics-certs\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.273619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.273591 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppndz\" (UniqueName: \"kubernetes.io/projected/64445549-ad95-4060-9522-ab6b550cecbf-kube-api-access-ppndz\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.336086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.336013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:08.448619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.448498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:08.787060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.772784 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:08.815058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:08.810940 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:09.035054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:09.034047 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:09.491977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:09.491939 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8"] Nov 04 00:02:09.498369 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:02:09.498331 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64445549_ad95_4060_9522_ab6b550cecbf.slice/crio-e1585b9636c7866b7dce82792b54fb243f39b4362563c8938da0a3132061d812 WatchSource:0}: Error finding container e1585b9636c7866b7dce82792b54fb243f39b4362563c8938da0a3132061d812: Status 404 returned error can't find the container with id e1585b9636c7866b7dce82792b54fb243f39b4362563c8938da0a3132061d812 Nov 04 00:02:09.782984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:09.782895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:10.306240 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:10.306197 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" event={"ID":"64445549-ad95-4060-9522-ab6b550cecbf","Type":"ContainerStarted","Data":"4c0e1021d5aede26f1ca6d88d379d72274e897e3e99309e9759f6fca3d36d7fb"} Nov 04 00:02:10.306412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:10.306250 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" event={"ID":"64445549-ad95-4060-9522-ab6b550cecbf","Type":"ContainerStarted","Data":"e1585b9636c7866b7dce82792b54fb243f39b4362563c8938da0a3132061d812"} Nov 04 00:02:10.306470 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:10.306449 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:10.341305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:10.341252 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" podStartSLOduration=1.828799449 podStartE2EDuration="2.341237911s" totalImagesPullingTime="512.438462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:02:10.336348533 +0000 UTC m=+1949.158944698" watchObservedRunningTime="2026-11-04 00:02:10.341237911 +0000 UTC m=+1949.163834072" Nov 04 00:02:10.879949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:10.879892 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:11.245468 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:11.245434 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:11.884005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:11.883384 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:12.068180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:12.068143 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:12.183926 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:12.183822 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:13.051262 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:13.051224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:13.415286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:13.415194 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:14.044694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:14.044659 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:15.011013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:15.008769 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:15.992197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:15.992142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:16.322181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:16.322102 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:02:16.970495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:16.970458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:18.075928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:18.075887 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:19.161948 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:19.161912 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:20.005573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:20.005532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:21.095706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:21.095662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:22.148124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:22.148087 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:23.160435 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:23.160347 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:23.465209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:23.465170 6208 generic.go:356] "Generic (PLEG): container finished" podID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerID="4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302" exitCode=0 Nov 04 00:02:23.465418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:23.465219 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" event={"ID":"4ad550b3-aad8-40e6-80b2-98d821384e5e","Type":"ContainerDied","Data":"4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302"} Nov 04 00:02:23.465768 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:23.465747 6208 scope.go:122] "RemoveContainer" containerID="4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302" Nov 04 00:02:23.467878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:23.467838 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:02:23.467997 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:23.467923 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:02:23.467997 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:23.467956 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:02:24.195988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:24.195615 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:24.351057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:24.350556 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:24.525054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:24.505086 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" event={"ID":"4ad550b3-aad8-40e6-80b2-98d821384e5e","Type":"ContainerStarted","Data":"6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374"} Nov 04 00:02:24.802104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:24.801694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:25.178360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:25.178257 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:02:25.179980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:25.179949 6208 patch_prober.go:29] interesting pod/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 00:02:25.180159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:25.180018 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 00:02:25.394395 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:25.394361 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:26.178551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:26.178507 6208 patch_prober.go:29] interesting pod/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 00:02:26.178781 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:26.178567 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 00:02:26.885658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:26.885624 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:27.178118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:27.177536 6208 patch_prober.go:29] interesting pod/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 00:02:27.178118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:27.177604 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 00:02:27.929491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:27.929453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:28.178579 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:28.178536 6208 patch_prober.go:29] interesting pod/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 00:02:28.178770 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:28.178610 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 00:02:29.011552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:29.011511 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:29.178558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:29.178521 6208 patch_prober.go:29] interesting pod/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 00:02:29.178777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:29.178588 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 00:02:29.484344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:29.484292 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:02:29.788760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:29.788668 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:30.088455 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:30.088326 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:30.180475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:30.180445 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:02:30.181570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:30.181549 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:02:31.150717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:31.150683 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:31.998001 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:31.997962 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:32.931571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:32.931532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:33.993487 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:33.993451 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:34.874917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:34.874882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:35.085319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:35.085280 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:35.707636 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:02:35.707590 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:02:35.830902 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:35.830862 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:36.097797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:36.097717 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:36.864687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:36.864655 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:37.415473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:37.415434 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:37.786162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:37.786119 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:38.435942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:38.435903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:38.880644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:38.880605 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:39.723054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:39.717491 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:39.851376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:39.851341 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:39.996736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:39.996634 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:40.135680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:40.135650 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:40.323567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:40.323483 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:40.489420 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:40.489381 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:40.668641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:40.668557 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:40.859533 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:40.859500 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:41.404116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:41.350401 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:41.524058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:41.520332 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:41.869056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:41.862978 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:42.189114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:42.188909 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:42.694060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:42.683655 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:43.032252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:43.032175 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:43.879058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:43.874201 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:44.524470 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:44.524435 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:45.114686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:45.114645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:46.045698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:46.045657 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:46.782346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:46.782311 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:47.635198 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:47.635157 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:47.795304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:47.795255 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:47.918693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:47.918574 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:48.376498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:48.376460 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:48.729734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:48.729686 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v"] Nov 04 00:02:48.730326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:48.729995 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" containerID="cri-o://20b58e7c93c99efba0c738e8ee7cf778c0cfec59bf5e867612186d439f33ca21" gracePeriod=30 Nov 04 00:02:48.758840 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:48.757628 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:48.953819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:48.953725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:50.192100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:50.191987 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:50.685382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:50.685328 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:51.482556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:51.482463 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:51.755773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:51.755638 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:52.000116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:52.000084 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:53.185515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:53.185472 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:53.312929 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:53.312897 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:53.860751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:53.860712 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:54.032593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:54.032506 6208 generic.go:356] "Generic (PLEG): container finished" podID="676b3179-ba28-4ce1-baa2-d3557794a632" containerID="20b58e7c93c99efba0c738e8ee7cf778c0cfec59bf5e867612186d439f33ca21" exitCode=0 Nov 04 00:02:54.032593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:54.032568 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" event={"ID":"676b3179-ba28-4ce1-baa2-d3557794a632","Type":"ContainerDied","Data":"20b58e7c93c99efba0c738e8ee7cf778c0cfec59bf5e867612186d439f33ca21"} Nov 04 00:02:54.032816 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:54.032665 6208 scope.go:122] "RemoveContainer" containerID="12fa0f85e44d5ef6bc2c8288efd2d317a883259b250a36bbca3eb2a42653454d" Nov 04 00:02:55.168116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:55.168079 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:55.190322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:55.190282 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:02:55.190518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:55.190369 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:02:55.711314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:55.711260 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:56.054637 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.051627 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:56.170858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.170822 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:02:56.366124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366013 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-kube-api-access-mgwpx\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-kube-api-access-mgwpx\") pod \"676b3179-ba28-4ce1-baa2-d3557794a632\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " Nov 04 00:02:56.366324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366121 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-certs\") pod \"676b3179-ba28-4ce1-baa2-d3557794a632\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " Nov 04 00:02:56.366324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366230 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-socket\") pod \"676b3179-ba28-4ce1-baa2-d3557794a632\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " Nov 04 00:02:56.366324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366269 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/676b3179-ba28-4ce1-baa2-d3557794a632-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676b3179-ba28-4ce1-baa2-d3557794a632-istio-podinfo\") pod \"676b3179-ba28-4ce1-baa2-d3557794a632\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " Nov 04 00:02:56.366467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366393 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-istio-token\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-istio-token\") pod \"676b3179-ba28-4ce1-baa2-d3557794a632\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " Nov 04 00:02:56.366467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366434 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-certs" pod "676b3179-ba28-4ce1-baa2-d3557794a632" (UID: "676b3179-ba28-4ce1-baa2-d3557794a632"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:02:56.366585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366491 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-envoy\") pod \"676b3179-ba28-4ce1-baa2-d3557794a632\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " Nov 04 00:02:56.366585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366515 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-socket" pod "676b3179-ba28-4ce1-baa2-d3557794a632" (UID: "676b3179-ba28-4ce1-baa2-d3557794a632"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:02:56.366585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366537 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-credential-socket\") pod \"676b3179-ba28-4ce1-baa2-d3557794a632\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " Nov 04 00:02:56.366793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366622 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676b3179-ba28-4ce1-baa2-d3557794a632-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676b3179-ba28-4ce1-baa2-d3557794a632-istiod-ca-cert\") pod \"676b3179-ba28-4ce1-baa2-d3557794a632\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " Nov 04 00:02:56.366793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366685 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-data\") pod \"676b3179-ba28-4ce1-baa2-d3557794a632\" (UID: \"676b3179-ba28-4ce1-baa2-d3557794a632\") " Nov 04 00:02:56.366793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.366769 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-credential-socket" pod "676b3179-ba28-4ce1-baa2-d3557794a632" (UID: "676b3179-ba28-4ce1-baa2-d3557794a632"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:02:56.367119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.367092 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676b3179-ba28-4ce1-baa2-d3557794a632-istiod-ca-cert" pod "676b3179-ba28-4ce1-baa2-d3557794a632" (UID: "676b3179-ba28-4ce1-baa2-d3557794a632"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:02:56.367226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.367162 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-data" pod "676b3179-ba28-4ce1-baa2-d3557794a632" (UID: "676b3179-ba28-4ce1-baa2-d3557794a632"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:02:56.367443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.367426 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:02:56.367494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.367449 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:02:56.367494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.367466 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:02:56.367494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.367480 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676b3179-ba28-4ce1-baa2-d3557794a632-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:02:56.367586 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.367494 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:02:56.390365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.390341 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/676b3179-ba28-4ce1-baa2-d3557794a632-istio-podinfo" pod "676b3179-ba28-4ce1-baa2-d3557794a632" (UID: "676b3179-ba28-4ce1-baa2-d3557794a632"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:02:56.390603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.390577 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-envoy" pod "676b3179-ba28-4ce1-baa2-d3557794a632" (UID: "676b3179-ba28-4ce1-baa2-d3557794a632"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:02:56.390700 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.390670 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-kube-api-access-mgwpx" pod "676b3179-ba28-4ce1-baa2-d3557794a632" (UID: "676b3179-ba28-4ce1-baa2-d3557794a632"). InnerVolumeSpecName "kube-api-access-mgwpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:02:56.390757 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.390695 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-istio-token" pod "676b3179-ba28-4ce1-baa2-d3557794a632" (UID: "676b3179-ba28-4ce1-baa2-d3557794a632"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:02:56.468764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.468727 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676b3179-ba28-4ce1-baa2-d3557794a632-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:02:56.468764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.468766 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwpx\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-kube-api-access-mgwpx\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:02:56.468955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.468783 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676b3179-ba28-4ce1-baa2-d3557794a632-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:02:56.468955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.468797 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676b3179-ba28-4ce1-baa2-d3557794a632-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:02:56.628841 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.628747 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:56.746046 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:56.745996 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:57.048954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:57.048921 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:57.105294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:57.105267 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" Nov 04 00:02:57.105497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:57.105306 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v" event={"ID":"676b3179-ba28-4ce1-baa2-d3557794a632","Type":"ContainerDied","Data":"86c39750f43376aef4a623abf3c723b01ba3a5f6362477f5ab41a628f1c0dce3"} Nov 04 00:02:57.105497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:57.105388 6208 scope.go:122] "RemoveContainer" containerID="20b58e7c93c99efba0c738e8ee7cf778c0cfec59bf5e867612186d439f33ca21" Nov 04 00:02:57.139913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:57.138380 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v"] Nov 04 00:02:57.145128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:57.144811 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-27f8a1-openshift-default-57ddd499cc-z2w6v"] Nov 04 00:02:57.304080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:57.303991 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" path="/var/lib/kubelet/pods/676b3179-ba28-4ce1-baa2-d3557794a632/volumes" Nov 04 00:02:57.308468 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:57.308428 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:57.441462 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:57.441423 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:58.079745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.079694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:58.223666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.223631 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt"] Nov 04 00:02:58.228069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.228028 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:02:58.247916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.247877 6208 cpu_manager.go:414] "containerMap: removing container" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" Nov 04 00:02:58.248153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.248121 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" Nov 04 00:02:58.248359 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.248343 6208 cpu_manager.go:414] "containerMap: removing container" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" Nov 04 00:02:58.248521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.248448 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" Nov 04 00:02:58.248930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.248909 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" Nov 04 00:02:58.249091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.249078 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="676b3179-ba28-4ce1-baa2-d3557794a632" containerName="istio-proxy" Nov 04 00:02:58.291698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.291104 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:02:58.308073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.307666 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-06ade4\"" Nov 04 00:02:58.312667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.312638 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-06ade4-dockercfg-d842v\"" Nov 04 00:02:58.440916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.440878 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt"] Nov 04 00:02:58.475618 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.475565 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62966127-1cff-4a1a-b9d5-16de42660432-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt\" (UID: \"62966127-1cff-4a1a-b9d5-16de42660432\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:02:58.478721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.478689 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw6br\" (UniqueName: \"kubernetes.io/projected/62966127-1cff-4a1a-b9d5-16de42660432-kube-api-access-pw6br\") pod \"payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt\" (UID: \"62966127-1cff-4a1a-b9d5-16de42660432\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:02:58.581667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.581626 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62966127-1cff-4a1a-b9d5-16de42660432-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt\" (UID: \"62966127-1cff-4a1a-b9d5-16de42660432\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:02:58.581934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.581712 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw6br\" (UniqueName: \"kubernetes.io/projected/62966127-1cff-4a1a-b9d5-16de42660432-kube-api-access-pw6br\") pod \"payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt\" (UID: \"62966127-1cff-4a1a-b9d5-16de42660432\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:02:58.582447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.582381 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62966127-1cff-4a1a-b9d5-16de42660432-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt\" (UID: \"62966127-1cff-4a1a-b9d5-16de42660432\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:02:58.601718 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.601685 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw6br\" (UniqueName: \"kubernetes.io/projected/62966127-1cff-4a1a-b9d5-16de42660432-kube-api-access-pw6br\") pod \"payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt\" (UID: \"62966127-1cff-4a1a-b9d5-16de42660432\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:02:58.765370 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:58.765329 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:02:59.110514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:59.110482 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt"] Nov 04 00:02:59.111545 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:02:59.111501 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62966127_1cff_4a1a_b9d5_16de42660432.slice/crio-b6814a2a6d494faea6a82058c973d506d7d715ad62715fb17055c05accf6f352 WatchSource:0}: Error finding container b6814a2a6d494faea6a82058c973d506d7d715ad62715fb17055c05accf6f352: Status 404 returned error can't find the container with id b6814a2a6d494faea6a82058c973d506d7d715ad62715fb17055c05accf6f352 Nov 04 00:02:59.131121 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:59.131075 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" event={"ID":"62966127-1cff-4a1a-b9d5-16de42660432","Type":"ContainerStarted","Data":"b6814a2a6d494faea6a82058c973d506d7d715ad62715fb17055c05accf6f352"} Nov 04 00:02:59.269144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:02:59.269113 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:00.140252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:00.140214 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" event={"ID":"62966127-1cff-4a1a-b9d5-16de42660432","Type":"ContainerStarted","Data":"0671c050e8723da33a720d98b163d06b30d579e6247146a63cf18a0f209ea583"} Nov 04 00:03:00.155379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:00.140294 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:03:00.160833 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:00.160786 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" podStartSLOduration=2.160769101 podStartE2EDuration="2.160769101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:03:00.15880985 +0000 UTC m=+1998.981406011" watchObservedRunningTime="2026-11-04 00:03:00.160769101 +0000 UTC m=+1998.983365263" Nov 04 00:03:00.546610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:00.536067 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:00.926469 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:00.926439 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 00:03:00.926683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:00.926503 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 00:03:01.149069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.148975 6208 generic.go:356] "Generic (PLEG): container finished" podID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerID="eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df" exitCode=0 Nov 04 00:03:01.149495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.149060 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" event={"ID":"f5979d19-2211-458a-9734-0a9a8ddfd617","Type":"ContainerDied","Data":"eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df"} Nov 04 00:03:01.149578 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.149559 6208 scope.go:122] "RemoveContainer" containerID="eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df" Nov 04 00:03:01.150795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.150776 6208 generic.go:356] "Generic (PLEG): container finished" podID="77b9470d-1121-4176-be18-bd5348ae17db" containerID="72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7" exitCode=0 Nov 04 00:03:01.150889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.150863 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" event={"ID":"77b9470d-1121-4176-be18-bd5348ae17db","Type":"ContainerDied","Data":"72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7"} Nov 04 00:03:01.151408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.151378 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:03:01.151408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.151403 6208 scope.go:122] "RemoveContainer" containerID="72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7" Nov 04 00:03:01.151550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.151452 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:03:01.151550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.151489 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:03:01.152962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.152936 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:03:01.153081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.153001 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:03:01.153081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.153050 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:03:01.588400 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:01.587088 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:02.159563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:02.159527 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" event={"ID":"f5979d19-2211-458a-9734-0a9a8ddfd617","Type":"ContainerStarted","Data":"55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70"} Nov 04 00:03:02.161443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:02.161418 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" event={"ID":"77b9470d-1121-4176-be18-bd5348ae17db","Type":"ContainerStarted","Data":"f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45"} Nov 04 00:03:02.323513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:02.323476 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:03:02.324284 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:02.324260 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 04 00:03:02.324370 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:02.324310 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 04 00:03:02.732134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:02.732105 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:03:02.732309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:02.732259 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 00:03:02.732359 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:02.732299 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 00:03:03.324232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:03.324198 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 04 00:03:03.324735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:03.324254 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 04 00:03:03.340965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:03.340934 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:03.732630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:03.732597 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 00:03:03.732823 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:03.732646 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 00:03:04.324087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:04.324020 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 04 00:03:04.324308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:04.324101 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 04 00:03:04.731733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:04.731687 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 00:03:04.731944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:04.731749 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 00:03:04.949390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:04.949279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:05.324686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:05.324652 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 04 00:03:05.325309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:05.325266 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 04 00:03:05.731949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:05.731906 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 00:03:05.732155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:05.731966 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 00:03:05.810113 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:03:05.809923 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:03:06.324144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:06.324109 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 04 00:03:06.324384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:06.324359 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 04 00:03:06.435086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:06.435049 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:06.732560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:06.732527 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 00:03:06.732751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:06.732580 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 00:03:06.931354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:06.931314 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:07.324396 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:07.324364 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 04 00:03:07.324573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:07.324411 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 04 00:03:07.623005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:07.622920 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z"] Nov 04 00:03:07.727066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:07.719860 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:07.732588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:07.732145 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 00:03:07.732588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:07.732229 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 00:03:07.826649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:07.826619 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:08.323794 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:08.323752 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 04 00:03:08.323980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:08.323829 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 04 00:03:08.345754 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:08.345723 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:08.737019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:08.736989 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:03:08.737518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:08.737419 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:03:08.738239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:08.738207 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:03:08.843934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:08.843902 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:03:08.961622 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:08.961585 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:09.329097 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:09.329061 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:03:09.330234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:09.330214 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:03:09.981715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:09.981682 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:11.154620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:11.154586 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:03:11.222311 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:11.222274 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:12.624313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:12.624251 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" podUID="a2177c20-6e16-48e7-948a-78fa2ce0fc66" containerName="payload-pre-processing" containerID="cri-o://05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f" gracePeriod=25 Nov 04 00:03:12.676823 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:12.676797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:12.933726 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:12.933695 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:03:12.942575 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:12.942544 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:13.034663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.034630 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2177c20-6e16-48e7-948a-78fa2ce0fc66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2177c20-6e16-48e7-948a-78fa2ce0fc66-plugins-config-volume\") pod \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\" (UID: \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\") " Nov 04 00:03:13.034842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.034710 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2177c20-6e16-48e7-948a-78fa2ce0fc66-kube-api-access-h4c2c\" (UniqueName: \"kubernetes.io/projected/a2177c20-6e16-48e7-948a-78fa2ce0fc66-kube-api-access-h4c2c\") pod \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\" (UID: \"a2177c20-6e16-48e7-948a-78fa2ce0fc66\") " Nov 04 00:03:13.037190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.035114 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2177c20-6e16-48e7-948a-78fa2ce0fc66-plugins-config-volume" pod "a2177c20-6e16-48e7-948a-78fa2ce0fc66" (UID: "a2177c20-6e16-48e7-948a-78fa2ce0fc66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:03:13.037190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.035502 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2177c20-6e16-48e7-948a-78fa2ce0fc66-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:13.037961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.037913 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2177c20-6e16-48e7-948a-78fa2ce0fc66-kube-api-access-h4c2c" pod "a2177c20-6e16-48e7-948a-78fa2ce0fc66" (UID: "a2177c20-6e16-48e7-948a-78fa2ce0fc66"). InnerVolumeSpecName "kube-api-access-h4c2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:03:13.137537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.137506 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4c2c\" (UniqueName: \"kubernetes.io/projected/a2177c20-6e16-48e7-948a-78fa2ce0fc66-kube-api-access-h4c2c\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:13.294462 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.294421 6208 generic.go:356] "Generic (PLEG): container finished" podID="a2177c20-6e16-48e7-948a-78fa2ce0fc66" containerID="05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f" exitCode=0 Nov 04 00:03:13.294642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.294503 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" Nov 04 00:03:13.296347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.296323 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" event={"ID":"a2177c20-6e16-48e7-948a-78fa2ce0fc66","Type":"ContainerDied","Data":"05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f"} Nov 04 00:03:13.296459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.296367 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z" event={"ID":"a2177c20-6e16-48e7-948a-78fa2ce0fc66","Type":"ContainerDied","Data":"9b03db8c1964a1ae22a20f85b6f1abc727f10bf45f241a28f28b881c0f24411f"} Nov 04 00:03:13.296459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.296394 6208 scope.go:122] "RemoveContainer" containerID="05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f" Nov 04 00:03:13.308918 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.308884 6208 scope.go:122] "RemoveContainer" containerID="05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f" Nov 04 00:03:13.309230 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:03:13.309208 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f\": container with ID starting with 05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f not found: ID does not exist" containerID="05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f" Nov 04 00:03:13.309294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.309237 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f"} err="failed to get container status \"05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f\": rpc error: code = NotFound desc = could not find container \"05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f\": container with ID starting with 05e0a4c7c6faec3a104ba9a853327b49d73d97cbe43ef36bebf73acfebc3781f not found: ID does not exist" Nov 04 00:03:13.325631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.325606 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z"] Nov 04 00:03:13.329468 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:13.329446 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18f5c0-6454559cf5-m5z2z"] Nov 04 00:03:14.050071 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:14.049999 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:14.149548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:14.149507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:14.291792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:14.291753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:14.546536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:14.546496 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:14.712375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:14.712338 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:14.948814 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:14.948723 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:15.296709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:15.296676 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2177c20-6e16-48e7-948a-78fa2ce0fc66" path="/var/lib/kubelet/pods/a2177c20-6e16-48e7-948a-78fa2ce0fc66/volumes" Nov 04 00:03:15.609329 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:15.609240 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:15.713901 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:15.713861 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:16.170642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.170605 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:16.282041 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.282005 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:16.536225 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.536142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:16.632768 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.632734 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:16.702904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.702868 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7"] Nov 04 00:03:16.704934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.704909 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a2177c20-6e16-48e7-948a-78fa2ce0fc66" containerName="payload-pre-processing" Nov 04 00:03:16.705140 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.705123 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2177c20-6e16-48e7-948a-78fa2ce0fc66" containerName="payload-pre-processing" Nov 04 00:03:16.705512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.705496 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2177c20-6e16-48e7-948a-78fa2ce0fc66" containerName="payload-pre-processing" Nov 04 00:03:16.710884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.710849 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.714947 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.714924 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2697a0\"/\"agc-cred-58159406142a\"" Nov 04 00:03:16.720641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.720613 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7"] Nov 04 00:03:16.799686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.799533 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.799686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.799581 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29r96\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-kube-api-access-29r96\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.799942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.799862 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.800070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.800022 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.901742 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.901703 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.901955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.901805 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.901955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.901930 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.902111 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.901966 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29r96\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-kube-api-access-29r96\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.902693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.902667 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.902835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.902812 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.904624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.904589 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:16.920730 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:16.919951 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29r96\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-kube-api-access-29r96\") pod \"payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:17.035794 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:17.035757 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:17.327136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:17.327098 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7"] Nov 04 00:03:17.340630 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:03:17.340584 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf49ee54_a7ff_4a6f_b66c_96811ec49894.slice/crio-293c3e99993dbff41cd85d04cbfedec8c1db89c8d5d56d61eed567453c7e5f9f WatchSource:0}: Error finding container 293c3e99993dbff41cd85d04cbfedec8c1db89c8d5d56d61eed567453c7e5f9f: Status 404 returned error can't find the container with id 293c3e99993dbff41cd85d04cbfedec8c1db89c8d5d56d61eed567453c7e5f9f Nov 04 00:03:17.368250 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:17.368220 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" event={"ID":"df49ee54-a7ff-4a6f-b66c-96811ec49894","Type":"ContainerStarted","Data":"293c3e99993dbff41cd85d04cbfedec8c1db89c8d5d56d61eed567453c7e5f9f"} Nov 04 00:03:18.382560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:18.382518 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" event={"ID":"df49ee54-a7ff-4a6f-b66c-96811ec49894","Type":"ContainerStarted","Data":"2ca081ae4c3561173fb6258ebd96d7ac99f478b3c96604cf15c8a73f66a56ea8"} Nov 04 00:03:18.382987 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:18.382687 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:18.402985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:18.402931 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" podStartSLOduration=2.402918101 podStartE2EDuration="2.402918101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:03:18.402168948 +0000 UTC m=+2017.224765115" watchObservedRunningTime="2026-11-04 00:03:18.402918101 +0000 UTC m=+2017.225514293" Nov 04 00:03:18.727907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:18.727868 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:20.337146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:20.337108 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:21.752252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:21.752218 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:23.065442 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:23.065406 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:24.399710 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:24.399675 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:26.028643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:26.028602 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:27.437089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:27.437050 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:29.094985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:29.094941 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:29.392913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:29.392828 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:03:29.458442 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:29.458405 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5"] Nov 04 00:03:30.204465 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.204429 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq"] Nov 04 00:03:30.210903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.210880 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.223223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.223198 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq"] Nov 04 00:03:30.225147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.225123 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-86deca-serving-cert\"" Nov 04 00:03:30.285963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.285932 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:30.389482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.389447 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:30.408937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.408904 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.409171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.409003 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7f005b-f784-4e32-9eee-6582056d16fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.409171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.409091 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286pl\" (UniqueName: \"kubernetes.io/projected/5a7f005b-f784-4e32-9eee-6582056d16fe-kube-api-access-286pl\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.409299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.409184 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-metrics-certs\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.491093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.491013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:30.510595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.510564 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-metrics-certs\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.510763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.510701 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.510813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.510794 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7f005b-f784-4e32-9eee-6582056d16fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.510904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.510890 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286pl\" (UniqueName: \"kubernetes.io/projected/5a7f005b-f784-4e32-9eee-6582056d16fe-kube-api-access-286pl\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.511545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.511515 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7f005b-f784-4e32-9eee-6582056d16fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.513101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.513083 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.513189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.513148 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-metrics-certs\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.523763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.523736 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-286pl\" (UniqueName: \"kubernetes.io/projected/5a7f005b-f784-4e32-9eee-6582056d16fe-kube-api-access-286pl\") pod \"maas-api-e2e-extmod-a-86deca-5746598546-gxrcq\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.533741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.533717 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:30.821145 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:30.821114 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq"] Nov 04 00:03:30.821496 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:03:30.821474 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7f005b_f784_4e32_9eee_6582056d16fe.slice/crio-745c8afcf16706eca3fc4fbdf2f7c2edb8bb4dd3c4a435fd8198bdd521880eb5 WatchSource:0}: Error finding container 745c8afcf16706eca3fc4fbdf2f7c2edb8bb4dd3c4a435fd8198bdd521880eb5: Status 404 returned error can't find the container with id 745c8afcf16706eca3fc4fbdf2f7c2edb8bb4dd3c4a435fd8198bdd521880eb5 Nov 04 00:03:31.500127 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:31.500089 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" event={"ID":"5a7f005b-f784-4e32-9eee-6582056d16fe","Type":"ContainerStarted","Data":"4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87"} Nov 04 00:03:31.500647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:31.500152 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" event={"ID":"5a7f005b-f784-4e32-9eee-6582056d16fe","Type":"ContainerStarted","Data":"745c8afcf16706eca3fc4fbdf2f7c2edb8bb4dd3c4a435fd8198bdd521880eb5"} Nov 04 00:03:31.500647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:31.500241 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:31.526661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:31.526603 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" podStartSLOduration=1.019662429 podStartE2EDuration="1.526582324s" totalImagesPullingTime="506.919895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:03:31.521627989 +0000 UTC m=+2030.344224157" watchObservedRunningTime="2026-11-04 00:03:31.526582324 +0000 UTC m=+2030.349178487" Nov 04 00:03:31.980572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:31.980532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:33.947976 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:33.947938 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:34.064331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.061667 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:34.261483 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.261436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:34.458773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.458730 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" podUID="77c3f135-3c06-451e-b048-ecc1fc1cf2e4" containerName="payload-processing" containerID="cri-o://6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac" gracePeriod=25 Nov 04 00:03:34.766610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.766586 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:03:34.827471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.825659 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-kube-api-access-2gj9q\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-kube-api-access-2gj9q\") pod \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " Nov 04 00:03:34.827471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.825762 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-routing-overlay-volume\") pod \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " Nov 04 00:03:34.827471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.825994 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-plugins-config-volume\") pod \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " Nov 04 00:03:34.827471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.826132 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-provider-credentials\") pod \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\" (UID: \"77c3f135-3c06-451e-b048-ecc1fc1cf2e4\") " Nov 04 00:03:34.828602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.828574 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-routing-overlay-volume" pod "77c3f135-3c06-451e-b048-ecc1fc1cf2e4" (UID: "77c3f135-3c06-451e-b048-ecc1fc1cf2e4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:03:34.835126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.829119 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-plugins-config-volume" pod "77c3f135-3c06-451e-b048-ecc1fc1cf2e4" (UID: "77c3f135-3c06-451e-b048-ecc1fc1cf2e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:03:34.876934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.876869 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-provider-credentials" pod "77c3f135-3c06-451e-b048-ecc1fc1cf2e4" (UID: "77c3f135-3c06-451e-b048-ecc1fc1cf2e4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:03:34.894204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.893848 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-kube-api-access-2gj9q" pod "77c3f135-3c06-451e-b048-ecc1fc1cf2e4" (UID: "77c3f135-3c06-451e-b048-ecc1fc1cf2e4"). InnerVolumeSpecName "kube-api-access-2gj9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:03:34.928913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.928870 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gj9q\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-kube-api-access-2gj9q\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:34.928913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.928912 6208 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-routing-overlay-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:34.929167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.928929 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:34.929167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:34.928943 6208 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77c3f135-3c06-451e-b048-ecc1fc1cf2e4-provider-credentials\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:35.505851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.505808 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:35.541427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.541376 6208 generic.go:356] "Generic (PLEG): container finished" podID="77c3f135-3c06-451e-b048-ecc1fc1cf2e4" containerID="6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac" exitCode=0 Nov 04 00:03:35.541882 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.541852 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" event={"ID":"77c3f135-3c06-451e-b048-ecc1fc1cf2e4","Type":"ContainerDied","Data":"6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac"} Nov 04 00:03:35.542128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.542110 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" event={"ID":"77c3f135-3c06-451e-b048-ecc1fc1cf2e4","Type":"ContainerDied","Data":"279f768e49fcc4f036e001e8edf480686a41e2eaff821fea4b135086e1168188"} Nov 04 00:03:35.542305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.542282 6208 scope.go:122] "RemoveContainer" containerID="6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac" Nov 04 00:03:35.542486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.542375 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5" Nov 04 00:03:35.581942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.581918 6208 scope.go:122] "RemoveContainer" containerID="6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac" Nov 04 00:03:35.582394 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:03:35.582365 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac\": container with ID starting with 6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac not found: ID does not exist" containerID="6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac" Nov 04 00:03:35.582496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.582407 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac"} err="failed to get container status \"6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac\": rpc error: code = NotFound desc = could not find container \"6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac\": container with ID starting with 6becfcfd872a228fed8a3bfd0552c907faae1377b96b810385a46817365c8dac not found: ID does not exist" Nov 04 00:03:35.588825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.588797 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5"] Nov 04 00:03:35.596275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:35.596241 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-798dhcks5"] Nov 04 00:03:35.706103 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:03:35.706067 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:03:36.765711 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:36.765677 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:37.296278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:37.296241 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c3f135-3c06-451e-b048-ecc1fc1cf2e4" path="/var/lib/kubelet/pods/77c3f135-3c06-451e-b048-ecc1fc1cf2e4/volumes" Nov 04 00:03:37.440282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:37.440240 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:37.522304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:37.522273 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:03:37.540101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:37.540064 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:39.151505 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:39.151468 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:40.607813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:40.607771 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:42.197973 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:42.197933 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:43.556476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:43.556434 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:43.683071 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:43.682977 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:43.815267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:43.815190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:44.345698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:44.345654 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:44.956428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:44.956395 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x"] Nov 04 00:03:44.956916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:44.956619 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" containerID="cri-o://6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374" gracePeriod=30 Nov 04 00:03:44.997004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:44.996934 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:45.187352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:45.186939 6208 patch_prober.go:29] interesting pod/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:03:45.187352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:45.187001 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:03:45.858210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:45.858166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:46.896919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:46.896883 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:47.036645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:47.036613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:47.492060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:47.483331 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:47.639122 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:47.638928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:49.239547 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:49.239510 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:50.245202 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.245136 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:03:50.352338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.352297 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-podinfo\") pod \"4ad550b3-aad8-40e6-80b2-98d821384e5e\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " Nov 04 00:03:50.352540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.352509 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad550b3-aad8-40e6-80b2-98d821384e5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad550b3-aad8-40e6-80b2-98d821384e5e-istiod-ca-cert\") pod \"4ad550b3-aad8-40e6-80b2-98d821384e5e\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " Nov 04 00:03:50.352625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.352554 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-token\") pod \"4ad550b3-aad8-40e6-80b2-98d821384e5e\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " Nov 04 00:03:50.352625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.352600 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-certs\") pod \"4ad550b3-aad8-40e6-80b2-98d821384e5e\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " Nov 04 00:03:50.352740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.352647 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-kube-api-access-cj2mr\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-kube-api-access-cj2mr\") pod \"4ad550b3-aad8-40e6-80b2-98d821384e5e\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " Nov 04 00:03:50.352796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.352778 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-credential-socket\") pod \"4ad550b3-aad8-40e6-80b2-98d821384e5e\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " Nov 04 00:03:50.354461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.352894 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-envoy\") pod \"4ad550b3-aad8-40e6-80b2-98d821384e5e\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " Nov 04 00:03:50.354461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.352969 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad550b3-aad8-40e6-80b2-98d821384e5e-istiod-ca-cert" pod "4ad550b3-aad8-40e6-80b2-98d821384e5e" (UID: "4ad550b3-aad8-40e6-80b2-98d821384e5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:03:50.354461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.353011 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-socket\") pod \"4ad550b3-aad8-40e6-80b2-98d821384e5e\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " Nov 04 00:03:50.354461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.353096 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-data\") pod \"4ad550b3-aad8-40e6-80b2-98d821384e5e\" (UID: \"4ad550b3-aad8-40e6-80b2-98d821384e5e\") " Nov 04 00:03:50.354461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.353278 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-credential-socket" pod "4ad550b3-aad8-40e6-80b2-98d821384e5e" (UID: "4ad550b3-aad8-40e6-80b2-98d821384e5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:03:50.354461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.353914 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad550b3-aad8-40e6-80b2-98d821384e5e-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:50.354461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.353938 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:50.354461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.354175 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-data" pod "4ad550b3-aad8-40e6-80b2-98d821384e5e" (UID: "4ad550b3-aad8-40e6-80b2-98d821384e5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:03:50.354461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.354402 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-socket" pod "4ad550b3-aad8-40e6-80b2-98d821384e5e" (UID: "4ad550b3-aad8-40e6-80b2-98d821384e5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:03:50.355378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.355289 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-certs" pod "4ad550b3-aad8-40e6-80b2-98d821384e5e" (UID: "4ad550b3-aad8-40e6-80b2-98d821384e5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:03:50.355515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.355493 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-token" pod "4ad550b3-aad8-40e6-80b2-98d821384e5e" (UID: "4ad550b3-aad8-40e6-80b2-98d821384e5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:03:50.355594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.355532 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-kube-api-access-cj2mr" pod "4ad550b3-aad8-40e6-80b2-98d821384e5e" (UID: "4ad550b3-aad8-40e6-80b2-98d821384e5e"). InnerVolumeSpecName "kube-api-access-cj2mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:03:50.355594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.355539 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-podinfo" pod "4ad550b3-aad8-40e6-80b2-98d821384e5e" (UID: "4ad550b3-aad8-40e6-80b2-98d821384e5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:03:50.358206 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.358173 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-envoy" pod "4ad550b3-aad8-40e6-80b2-98d821384e5e" (UID: "4ad550b3-aad8-40e6-80b2-98d821384e5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:03:50.455137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.455099 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:50.455137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.455132 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:50.455137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.455144 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj2mr\" (UniqueName: \"kubernetes.io/projected/4ad550b3-aad8-40e6-80b2-98d821384e5e-kube-api-access-cj2mr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:50.455398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.455153 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:50.455398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.455163 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:50.455398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.455171 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:50.455398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.455179 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad550b3-aad8-40e6-80b2-98d821384e5e-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:03:50.723090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.723050 6208 generic.go:356] "Generic (PLEG): container finished" podID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerID="6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374" exitCode=0 Nov 04 00:03:50.723309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.723108 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" Nov 04 00:03:50.723309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.723129 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" event={"ID":"4ad550b3-aad8-40e6-80b2-98d821384e5e","Type":"ContainerDied","Data":"6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374"} Nov 04 00:03:50.723309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.723199 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x" event={"ID":"4ad550b3-aad8-40e6-80b2-98d821384e5e","Type":"ContainerDied","Data":"c5c2a4d2ba777701bca6a024d5ee59cbd2d1e10177ab9ad070fe231ab3f1fd68"} Nov 04 00:03:50.723309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.723228 6208 scope.go:122] "RemoveContainer" containerID="6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374" Nov 04 00:03:50.737154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.737133 6208 scope.go:122] "RemoveContainer" containerID="4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302" Nov 04 00:03:50.751283 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.751253 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x"] Nov 04 00:03:50.752618 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.752590 6208 scope.go:122] "RemoveContainer" containerID="6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374" Nov 04 00:03:50.752995 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:03:50.752963 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374\": container with ID starting with 6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374 not found: ID does not exist" containerID="6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374" Nov 04 00:03:50.753128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.753003 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374"} err="failed to get container status \"6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374\": rpc error: code = NotFound desc = could not find container \"6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374\": container with ID starting with 6548a58e639038fcf20a5b43db38ff302285127d54b8c8c47ce1b553b203f374 not found: ID does not exist" Nov 04 00:03:50.753128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.753023 6208 scope.go:122] "RemoveContainer" containerID="4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302" Nov 04 00:03:50.753347 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:03:50.753327 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302\": container with ID starting with 4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302 not found: ID does not exist" containerID="4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302" Nov 04 00:03:50.753414 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.753353 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302"} err="failed to get container status \"4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302\": rpc error: code = NotFound desc = could not find container \"4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302\": container with ID starting with 4048247b8fbd2cf5e0244d509f11475d62c00412e9cc656c4def713a6a238302 not found: ID does not exist" Nov 04 00:03:50.755945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.755923 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-18f5c0-openshift-default-8696785665-rsm2x"] Nov 04 00:03:50.827224 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:50.827180 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:51.265152 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:51.265119 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:51.296630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:51.296598 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" path="/var/lib/kubelet/pods/4ad550b3-aad8-40e6-80b2-98d821384e5e/volumes" Nov 04 00:03:51.379904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:51.379870 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:52.592294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.592254 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:52.631343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.631313 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x"] Nov 04 00:03:52.631971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.631958 6208 cpu_manager.go:414] "containerMap: removing container" podUID="77c3f135-3c06-451e-b048-ecc1fc1cf2e4" containerName="payload-processing" Nov 04 00:03:52.632061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.631973 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c3f135-3c06-451e-b048-ecc1fc1cf2e4" containerName="payload-processing" Nov 04 00:03:52.632061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.631995 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" Nov 04 00:03:52.632061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.632001 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" Nov 04 00:03:52.632061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.632007 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" Nov 04 00:03:52.632061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.632012 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" Nov 04 00:03:52.632218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.632164 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c3f135-3c06-451e-b048-ecc1fc1cf2e4" containerName="payload-processing" Nov 04 00:03:52.632218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.632178 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" Nov 04 00:03:52.632218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.632183 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad550b3-aad8-40e6-80b2-98d821384e5e" containerName="istio-proxy" Nov 04 00:03:52.637118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.637095 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:03:52.644212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.644186 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-71b2f3\"" Nov 04 00:03:52.646077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.646052 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x"] Nov 04 00:03:52.677278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.677237 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt"] Nov 04 00:03:52.689476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.689434 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc3bde-6478-4a87-af40-513329416597-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x\" (UID: \"cabc3bde-6478-4a87-af40-513329416597\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:03:52.689666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.689495 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwn7s\" (UniqueName: \"kubernetes.io/projected/cabc3bde-6478-4a87-af40-513329416597-kube-api-access-cwn7s\") pod \"payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x\" (UID: \"cabc3bde-6478-4a87-af40-513329416597\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:03:52.707651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.707614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:52.709256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.709227 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt"] Nov 04 00:03:52.709371 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.709348 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:03:52.715777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.715753 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-71b2f3-dockercfg-r54fc\"" Nov 04 00:03:52.823090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.820744 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt\" (UID: \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:03:52.823090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.820891 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc3bde-6478-4a87-af40-513329416597-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x\" (UID: \"cabc3bde-6478-4a87-af40-513329416597\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:03:52.823090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.820925 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwn7s\" (UniqueName: \"kubernetes.io/projected/cabc3bde-6478-4a87-af40-513329416597-kube-api-access-cwn7s\") pod \"payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x\" (UID: \"cabc3bde-6478-4a87-af40-513329416597\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:03:52.823090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.820955 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6c84\" (UniqueName: \"kubernetes.io/projected/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-kube-api-access-c6c84\") pod \"payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt\" (UID: \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:03:52.860111 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.859401 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc3bde-6478-4a87-af40-513329416597-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x\" (UID: \"cabc3bde-6478-4a87-af40-513329416597\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:03:52.904054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.896642 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwn7s\" (UniqueName: \"kubernetes.io/projected/cabc3bde-6478-4a87-af40-513329416597-kube-api-access-cwn7s\") pod \"payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x\" (UID: \"cabc3bde-6478-4a87-af40-513329416597\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:03:52.929058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.928192 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt\" (UID: \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:03:52.929058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.928393 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6c84\" (UniqueName: \"kubernetes.io/projected/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-kube-api-access-c6c84\") pod \"payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt\" (UID: \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:03:52.936069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.929993 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt\" (UID: \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:03:52.976053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:52.961470 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:03:53.004489 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.004393 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6c84\" (UniqueName: \"kubernetes.io/projected/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-kube-api-access-c6c84\") pod \"payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt\" (UID: \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:03:53.027764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.027731 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:03:53.321285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.321256 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x"] Nov 04 00:03:53.321876 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:03:53.321855 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabc3bde_6478_4a87_af40_513329416597.slice/crio-1205141cb1c52501936752dab10905d8646ba82f1d7f29dba2eed2b3c5c9421b WatchSource:0}: Error finding container 1205141cb1c52501936752dab10905d8646ba82f1d7f29dba2eed2b3c5c9421b: Status 404 returned error can't find the container with id 1205141cb1c52501936752dab10905d8646ba82f1d7f29dba2eed2b3c5c9421b Nov 04 00:03:53.354201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.354174 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt"] Nov 04 00:03:53.358347 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:03:53.358319 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fe6082_38e3_4e74_8712_4a1e9977cfaa.slice/crio-9dc946e902427d6a00f552edbbed25e0032dd0420bdba5456422c4d3e488c3a1 WatchSource:0}: Error finding container 9dc946e902427d6a00f552edbbed25e0032dd0420bdba5456422c4d3e488c3a1: Status 404 returned error can't find the container with id 9dc946e902427d6a00f552edbbed25e0032dd0420bdba5456422c4d3e488c3a1 Nov 04 00:03:53.397943 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.397916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:53.753189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.753148 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" event={"ID":"cabc3bde-6478-4a87-af40-513329416597","Type":"ContainerStarted","Data":"5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a"} Nov 04 00:03:53.753645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.753220 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:03:53.753645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.753238 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" event={"ID":"cabc3bde-6478-4a87-af40-513329416597","Type":"ContainerStarted","Data":"1205141cb1c52501936752dab10905d8646ba82f1d7f29dba2eed2b3c5c9421b"} Nov 04 00:03:53.755463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.755435 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" event={"ID":"d2fe6082-38e3-4e74-8712-4a1e9977cfaa","Type":"ContainerStarted","Data":"e9aa6728475ec95d3e4142fa68c72c7a79c338e11ff6604a9162cff9948c5c8a"} Nov 04 00:03:53.755607 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.755486 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" event={"ID":"d2fe6082-38e3-4e74-8712-4a1e9977cfaa","Type":"ContainerStarted","Data":"9dc946e902427d6a00f552edbbed25e0032dd0420bdba5456422c4d3e488c3a1"} Nov 04 00:03:53.755607 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.755516 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:03:53.776313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.776256 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" podStartSLOduration=1.776242691 podStartE2EDuration="1.776242691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:03:53.771921364 +0000 UTC m=+2052.594517530" watchObservedRunningTime="2026-11-04 00:03:53.776242691 +0000 UTC m=+2052.598838877" Nov 04 00:03:53.792708 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:53.792660 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" podStartSLOduration=1.792649607 podStartE2EDuration="1.792649607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:03:53.79053434 +0000 UTC m=+2052.613130502" watchObservedRunningTime="2026-11-04 00:03:53.792649607 +0000 UTC m=+2052.615245769" Nov 04 00:03:54.004624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:54.004536 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:55.942104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:55.942070 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:03:56.702608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:56.702566 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7"] Nov 04 00:03:59.401963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:03:59.401917 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:00.906926 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:00.906885 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:01.703773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:01.703681 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" podUID="df49ee54-a7ff-4a6f-b66c-96811ec49894" containerName="payload-processing" containerID="cri-o://2ca081ae4c3561173fb6258ebd96d7ac99f478b3c96604cf15c8a73f66a56ea8" gracePeriod=25 Nov 04 00:04:01.828851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:01.828822 6208 generic.go:356] "Generic (PLEG): container finished" podID="df49ee54-a7ff-4a6f-b66c-96811ec49894" containerID="2ca081ae4c3561173fb6258ebd96d7ac99f478b3c96604cf15c8a73f66a56ea8" exitCode=0 Nov 04 00:04:01.828988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:01.828897 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" event={"ID":"df49ee54-a7ff-4a6f-b66c-96811ec49894","Type":"ContainerDied","Data":"2ca081ae4c3561173fb6258ebd96d7ac99f478b3c96604cf15c8a73f66a56ea8"} Nov 04 00:04:01.956386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:01.956327 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:01.984013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:01.983986 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:04:02.129316 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.129277 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-plugins-config-volume\") pod \"df49ee54-a7ff-4a6f-b66c-96811ec49894\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " Nov 04 00:04:02.129501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.129382 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-routing-overlay-volume\") pod \"df49ee54-a7ff-4a6f-b66c-96811ec49894\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " Nov 04 00:04:02.129501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.129459 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-provider-credentials\") pod \"df49ee54-a7ff-4a6f-b66c-96811ec49894\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " Nov 04 00:04:02.129501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.129487 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-kube-api-access-29r96\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-kube-api-access-29r96\") pod \"df49ee54-a7ff-4a6f-b66c-96811ec49894\" (UID: \"df49ee54-a7ff-4a6f-b66c-96811ec49894\") " Nov 04 00:04:02.129775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.129749 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-plugins-config-volume" pod "df49ee54-a7ff-4a6f-b66c-96811ec49894" (UID: "df49ee54-a7ff-4a6f-b66c-96811ec49894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:04:02.129868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.129812 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-routing-overlay-volume" pod "df49ee54-a7ff-4a6f-b66c-96811ec49894" (UID: "df49ee54-a7ff-4a6f-b66c-96811ec49894"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:04:02.130316 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.130295 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:04:02.130386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.130322 6208 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df49ee54-a7ff-4a6f-b66c-96811ec49894-routing-overlay-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:04:02.131697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.131674 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-provider-credentials" pod "df49ee54-a7ff-4a6f-b66c-96811ec49894" (UID: "df49ee54-a7ff-4a6f-b66c-96811ec49894"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:04:02.131775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.131729 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-kube-api-access-29r96" pod "df49ee54-a7ff-4a6f-b66c-96811ec49894" (UID: "df49ee54-a7ff-4a6f-b66c-96811ec49894"). InnerVolumeSpecName "kube-api-access-29r96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:04:02.232365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.232271 6208 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-provider-credentials\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:04:02.232365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.232308 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29r96\" (UniqueName: \"kubernetes.io/projected/df49ee54-a7ff-4a6f-b66c-96811ec49894-kube-api-access-29r96\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:04:02.837849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.837820 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" Nov 04 00:04:02.838064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.837816 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7" event={"ID":"df49ee54-a7ff-4a6f-b66c-96811ec49894","Type":"ContainerDied","Data":"293c3e99993dbff41cd85d04cbfedec8c1db89c8d5d56d61eed567453c7e5f9f"} Nov 04 00:04:02.838064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.837959 6208 scope.go:122] "RemoveContainer" containerID="2ca081ae4c3561173fb6258ebd96d7ac99f478b3c96604cf15c8a73f66a56ea8" Nov 04 00:04:02.864762 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.864731 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7"] Nov 04 00:04:02.868425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:02.868402 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2697a0/payload-processing-external-model-e2e-extmod-b-2697a0-7d982fvs7"] Nov 04 00:04:03.296062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:03.295997 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df49ee54-a7ff-4a6f-b66c-96811ec49894" path="/var/lib/kubelet/pods/df49ee54-a7ff-4a6f-b66c-96811ec49894/volumes" Nov 04 00:04:03.649831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:03.649705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:04.765958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:04.765929 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:04:04.766476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:04.766002 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:04:05.707753 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:04:05.707703 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:04:06.829211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:06.829170 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:06.929203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:06.929161 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:08.245003 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:08.244964 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:09.964121 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:09.964080 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:11.479173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:11.477514 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:12.841168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:12.841137 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:14.040698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:14.040657 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:14.140676 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:14.140631 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:16.523624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:16.523583 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:17.536254 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:17.536221 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:17.996245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:17.996193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:18.482170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:18.482137 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:18.888593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:18.888502 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:19.468485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.468447 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft"] Nov 04 00:04:19.486104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.486062 6208 cpu_manager.go:414] "containerMap: removing container" podUID="df49ee54-a7ff-4a6f-b66c-96811ec49894" containerName="payload-processing" Nov 04 00:04:19.486339 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.486321 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df49ee54-a7ff-4a6f-b66c-96811ec49894" containerName="payload-processing" Nov 04 00:04:19.486649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.486635 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="df49ee54-a7ff-4a6f-b66c-96811ec49894" containerName="payload-processing" Nov 04 00:04:19.506302 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.506265 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:19.521171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.519018 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.534904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.534876 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b4376d-openshift-default-dockercfg-ps4xc\"" Nov 04 00:04:19.536287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.536247 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-data\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.536497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.536465 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istiod-ca-cert\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.536710 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.536674 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwzh\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-kube-api-access-vzwzh\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.536993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.536972 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-socket\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.537226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.537200 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-certs\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.537390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.537363 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-podinfo\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.537629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.537600 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-envoy\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.537993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.537971 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-token\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.538702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.538679 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-credential-socket\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.566299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.560431 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft"] Nov 04 00:04:19.640017 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.639979 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-token\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640108 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-credential-socket\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640181 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-data\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640208 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istiod-ca-cert\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640236 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwzh\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-kube-api-access-vzwzh\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640300 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-socket\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640414 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-certs\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640458 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-podinfo\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640576 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-envoy\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640868 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-socket\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.640986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640899 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-certs\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.641126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.640991 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istiod-ca-cert\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.641279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.641255 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-data\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.641474 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.641448 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-credential-socket\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.643479 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.643443 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-envoy\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.644088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.644066 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-podinfo\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.666016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.665920 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-token\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.666332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.666016 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwzh\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-kube-api-access-vzwzh\") pod \"e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.705779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.705706 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:19.826233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.826138 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:19.879532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.879493 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:19.997195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:19.996942 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:20.171502 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:20.171112 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:20.290074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:20.289998 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:20.332627 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:20.332598 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft"] Nov 04 00:04:20.332918 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:04:20.332894 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9fe7e3_20d0_4b79_b147_ba9f6fb319fb.slice/crio-8fc5b76f13a090b033d57eba0cd0b77859d588463891d51fb7754d56338de2f5 WatchSource:0}: Error finding container 8fc5b76f13a090b033d57eba0cd0b77859d588463891d51fb7754d56338de2f5: Status 404 returned error can't find the container with id 8fc5b76f13a090b033d57eba0cd0b77859d588463891d51fb7754d56338de2f5 Nov 04 00:04:20.335846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:20.335806 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:04:20.335939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:20.335877 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:04:20.335939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:20.335913 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:04:20.432215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:20.432158 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:20.565859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:20.565822 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:21.014248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:21.014205 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" event={"ID":"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb","Type":"ContainerStarted","Data":"84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271"} Nov 04 00:04:21.014710 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:21.014269 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" event={"ID":"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb","Type":"ContainerStarted","Data":"8fc5b76f13a090b033d57eba0cd0b77859d588463891d51fb7754d56338de2f5"} Nov 04 00:04:21.063082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:21.063012 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" podStartSLOduration=2.062997593 podStartE2EDuration="2.062997593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:04:21.061275398 +0000 UTC m=+2079.883871560" watchObservedRunningTime="2026-11-04 00:04:21.062997593 +0000 UTC m=+2079.885593817" Nov 04 00:04:21.161313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:21.161276 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:21.319112 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:21.309486 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:21.813756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:21.813722 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:21.882149 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:21.882113 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:21.889245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:21.889213 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:22.068789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:22.065087 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:22.100318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:22.100290 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:04:23.004858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:23.004824 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:23.740760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:23.740724 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:23.845239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:23.845206 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:24.781857 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:24.781821 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:25.050924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.050818 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:25.132070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.132015 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79"] Nov 04 00:04:25.137963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.137933 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:25.143977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.143949 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79"] Nov 04 00:04:25.169349 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.169320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:25.226510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.226464 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b4376d\"" Nov 04 00:04:25.250515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.250489 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b4376d-dockercfg-5kq5s\"" Nov 04 00:04:25.312125 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.303250 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4acee3-e8e1-4257-90bd-8aab3f96e757-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79\" (UID: \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:25.312125 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.303317 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz5n9\" (UniqueName: \"kubernetes.io/projected/2d4acee3-e8e1-4257-90bd-8aab3f96e757-kube-api-access-dz5n9\") pod \"payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79\" (UID: \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:25.405570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.405538 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4acee3-e8e1-4257-90bd-8aab3f96e757-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79\" (UID: \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:25.406887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.406845 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz5n9\" (UniqueName: \"kubernetes.io/projected/2d4acee3-e8e1-4257-90bd-8aab3f96e757-kube-api-access-dz5n9\") pod \"payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79\" (UID: \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:25.407540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.406761 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4acee3-e8e1-4257-90bd-8aab3f96e757-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79\" (UID: \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:25.421467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.421440 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz5n9\" (UniqueName: \"kubernetes.io/projected/2d4acee3-e8e1-4257-90bd-8aab3f96e757-kube-api-access-dz5n9\") pod \"payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79\" (UID: \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:25.458962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.458919 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:25.764781 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:25.764744 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79"] Nov 04 00:04:25.765074 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:04:25.765006 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4acee3_e8e1_4257_90bd_8aab3f96e757.slice/crio-5d69872bddd2d0344531948410bc3f148e8053364cac6c45592e2b2c6ded1c8c WatchSource:0}: Error finding container 5d69872bddd2d0344531948410bc3f148e8053364cac6c45592e2b2c6ded1c8c: Status 404 returned error can't find the container with id 5d69872bddd2d0344531948410bc3f148e8053364cac6c45592e2b2c6ded1c8c Nov 04 00:04:26.082063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:26.081960 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:26.096865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:26.096827 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" event={"ID":"2d4acee3-e8e1-4257-90bd-8aab3f96e757","Type":"ContainerStarted","Data":"2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3"} Nov 04 00:04:26.097049 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:26.096892 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:26.097049 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:26.096909 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" event={"ID":"2d4acee3-e8e1-4257-90bd-8aab3f96e757","Type":"ContainerStarted","Data":"5d69872bddd2d0344531948410bc3f148e8053364cac6c45592e2b2c6ded1c8c"} Nov 04 00:04:26.139877 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:26.139813 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" podStartSLOduration=1.139792679 podStartE2EDuration="1.139792679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:04:26.137766184 +0000 UTC m=+2084.960362347" watchObservedRunningTime="2026-11-04 00:04:26.139792679 +0000 UTC m=+2084.962388845" Nov 04 00:04:27.671227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:27.671183 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:29.274797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:29.274757 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:30.609388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:30.609348 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:30.709828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:30.709775 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:31.597094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:31.597058 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:33.058780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:33.058750 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:33.178226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:33.178189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:33.321545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:33.321453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:34.487025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:34.486987 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:35.708338 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:04:35.708300 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:04:36.038524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:36.038436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:37.098374 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:37.098326 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:37.107802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:37.107778 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:04:37.539860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:37.539825 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:38.156645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:38.156597 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:39.600375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:39.600325 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:42.287216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:42.287188 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:04:42.338294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:42.338264 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:04:42.371239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:42.371206 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:43.958276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:43.958238 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:45.430963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:45.430930 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:47.310886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:47.310824 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:48.809228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:48.809190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:50.158708 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:50.158668 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:51.163561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:51.163511 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:53.234673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:53.234639 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:55.166264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:55.166219 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:57.496170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:57.496134 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:58.310204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:58.310166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:04:59.606884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:04:59.606845 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:01.444453 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:01.444419 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:02.864942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:02.864903 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq"] Nov 04 00:05:02.869561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:02.869542 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:02.877172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:02.877148 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b4376d-serving-cert\"" Nov 04 00:05:02.883334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:02.883308 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq"] Nov 04 00:05:02.939510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:02.939478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:02.985163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:02.985113 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2r5\" (UniqueName: \"kubernetes.io/projected/da7d10bd-27ea-429e-8730-1b67bb6e7e46-kube-api-access-zr2r5\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:02.985358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:02.985273 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:02.985415 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:02.985385 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d10bd-27ea-429e-8730-1b67bb6e7e46-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:02.985598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:02.985579 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-metrics-certs\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.039572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.039533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:03.092869 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.092823 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-metrics-certs\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.093809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.093776 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2r5\" (UniqueName: \"kubernetes.io/projected/da7d10bd-27ea-429e-8730-1b67bb6e7e46-kube-api-access-zr2r5\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.094143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.094119 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.094354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.094337 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d10bd-27ea-429e-8730-1b67bb6e7e46-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.102055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.095435 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d10bd-27ea-429e-8730-1b67bb6e7e46-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.105755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.105699 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.115156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.114070 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-metrics-certs\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.115889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.115859 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2r5\" (UniqueName: \"kubernetes.io/projected/da7d10bd-27ea-429e-8730-1b67bb6e7e46-kube-api-access-zr2r5\") pod \"maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.189685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.189648 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:03.482981 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.482938 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq"] Nov 04 00:05:03.484069 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:05:03.484024 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7d10bd_27ea_429e_8730_1b67bb6e7e46.slice/crio-7a023ad64c1609af3e6b6f85a74cca8cf24b3dda0e29f96d3b608a46201289af WatchSource:0}: Error finding container 7a023ad64c1609af3e6b6f85a74cca8cf24b3dda0e29f96d3b608a46201289af: Status 404 returned error can't find the container with id 7a023ad64c1609af3e6b6f85a74cca8cf24b3dda0e29f96d3b608a46201289af Nov 04 00:05:03.620525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:03.620476 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:04.450503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:04.450408 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" event={"ID":"da7d10bd-27ea-429e-8730-1b67bb6e7e46","Type":"ContainerStarted","Data":"4c0ace1f1efc9c0e7c762eeb379142967c6489cf117281650e998f8d58439623"} Nov 04 00:05:04.450503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:04.450474 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:04.450503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:04.450499 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" event={"ID":"da7d10bd-27ea-429e-8730-1b67bb6e7e46","Type":"ContainerStarted","Data":"7a023ad64c1609af3e6b6f85a74cca8cf24b3dda0e29f96d3b608a46201289af"} Nov 04 00:05:04.484694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:04.484644 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" podStartSLOduration=1.9063754560000001 podStartE2EDuration="2.484629732s" totalImagesPullingTime="578.254276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:05:04.481462152 +0000 UTC m=+2123.304058324" watchObservedRunningTime="2026-11-04 00:05:04.484629732 +0000 UTC m=+2123.307225894" Nov 04 00:05:04.512886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:04.512850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:05.168825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:05.168791 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:05.706900 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:05.706856 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:05:06.950581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:06.950548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:07.211395 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:07.211357 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:09.448109 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:09.448076 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:09.947246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:09.947209 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:10.168716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:10.168678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:10.464905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:10.464873 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:05:11.430758 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:11.430726 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:11.718481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:11.718454 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:12.623669 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.623637 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:12.729334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.729302 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:12.828268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.828231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:12.902230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.902157 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2"] Nov 04 00:05:12.906839 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.906816 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:12.910862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.910813 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"agc-cred-f9c874ea3ea4\"" Nov 04 00:05:12.912344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.912320 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-e2e--7c418871-dockercfg-jhtpz\"" Nov 04 00:05:12.912474 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.912459 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"routing-overlay\"" Nov 04 00:05:12.912709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.912656 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"kube-root-ca.crt\"" Nov 04 00:05:12.912709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.912465 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3a002b\"" Nov 04 00:05:12.914604 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.914581 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"openshift-service-ca.crt\"" Nov 04 00:05:12.918072 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:12.918049 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2"] Nov 04 00:05:13.019637 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.019593 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.019829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.019698 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.019829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.019797 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpddm\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-kube-api-access-bpddm\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.019925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.019825 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.129056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.115096 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q"] Nov 04 00:05:13.129056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.121801 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.129056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.121896 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.129056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.121993 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpddm\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-kube-api-access-bpddm\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.129056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.122020 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.129056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.127470 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.129056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.128234 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.129056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.128930 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.152522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.151691 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.152522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.152268 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1b5fca\"" Nov 04 00:05:13.152522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.152349 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-d87e354e0f00\"" Nov 04 00:05:13.152522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.152452 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-e2e--0bada0ae-dockercfg-hsvcw\"" Nov 04 00:05:13.152522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.152508 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-ba9e5343dcd6\"" Nov 04 00:05:13.152909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.152531 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"kube-root-ca.crt\"" Nov 04 00:05:13.152909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.152593 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"openshift-service-ca.crt\"" Nov 04 00:05:13.152909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.152601 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"routing-overlay\"" Nov 04 00:05:13.152909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.152450 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-261bd6139428\"" Nov 04 00:05:13.172202 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.171998 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q"] Nov 04 00:05:13.257860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.257826 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2x4r\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-kube-api-access-q2x4r\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.258070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.258006 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.258163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.258143 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.258233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.258214 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.262668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.262637 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpddm\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-kube-api-access-bpddm\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.360098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.360064 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2x4r\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-kube-api-access-q2x4r\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.360285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.360210 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.360285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.360262 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.360388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.360330 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.361090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.361029 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.361230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.361123 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.363119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.363097 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.372318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.372296 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2x4r\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-kube-api-access-q2x4r\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.525063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.524989 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:13.602766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.602012 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:13.852094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.852060 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2"] Nov 04 00:05:13.852627 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:05:13.852603 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39f9edd_f183_4e45_aa47_1eed0e9b5f8c.slice/crio-a60898e744e2c943741f130e80cd097a288b5bd0910da27bea3119a439ef1bf2 WatchSource:0}: Error finding container a60898e744e2c943741f130e80cd097a288b5bd0910da27bea3119a439ef1bf2: Status 404 returned error can't find the container with id a60898e744e2c943741f130e80cd097a288b5bd0910da27bea3119a439ef1bf2 Nov 04 00:05:13.920553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:13.920523 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q"] Nov 04 00:05:13.921226 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:05:13.921202 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2218fc_0055_4894_a3cf_4864b5a316af.slice/crio-3dcf361ddfca2cffe3f2af2b1416a066a399edac35ffa3815f35b9cefa51f8c3 WatchSource:0}: Error finding container 3dcf361ddfca2cffe3f2af2b1416a066a399edac35ffa3815f35b9cefa51f8c3: Status 404 returned error can't find the container with id 3dcf361ddfca2cffe3f2af2b1416a066a399edac35ffa3815f35b9cefa51f8c3 Nov 04 00:05:14.099146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.099020 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:14.530456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.530420 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" event={"ID":"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c","Type":"ContainerStarted","Data":"6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725"} Nov 04 00:05:14.530680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.530481 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" event={"ID":"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c","Type":"ContainerStarted","Data":"a60898e744e2c943741f130e80cd097a288b5bd0910da27bea3119a439ef1bf2"} Nov 04 00:05:14.530680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.530509 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:14.532192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.532160 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" event={"ID":"8d2218fc-0055-4894-a3cf-4864b5a316af","Type":"ContainerStarted","Data":"3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d"} Nov 04 00:05:14.532336 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.532202 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" event={"ID":"8d2218fc-0055-4894-a3cf-4864b5a316af","Type":"ContainerStarted","Data":"3dcf361ddfca2cffe3f2af2b1416a066a399edac35ffa3815f35b9cefa51f8c3"} Nov 04 00:05:14.532336 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.532221 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:14.550412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.550363 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" podStartSLOduration=2.550349767 podStartE2EDuration="2.550349767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:05:14.549655027 +0000 UTC m=+2133.372251194" watchObservedRunningTime="2026-11-04 00:05:14.550349767 +0000 UTC m=+2133.372945960" Nov 04 00:05:14.573988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.573932 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" podStartSLOduration=1.573913456 podStartE2EDuration="1.573913456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:05:14.569968432 +0000 UTC m=+2133.392564595" watchObservedRunningTime="2026-11-04 00:05:14.573913456 +0000 UTC m=+2133.396509620" Nov 04 00:05:14.924586 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:14.924468 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:15.342981 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:15.342939 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2"] Nov 04 00:05:15.387683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:15.387645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:15.596108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:15.593592 6208 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" secret="" err="secret \"payload-processing-external-model-e2e--7c418871-dockercfg-jhtpz\" not found" Nov 04 00:05:15.725056 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:15.722945 6208 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a002b/agc-cred-f9c874ea3ea4: secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:15.725056 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:15.722974 6208 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2: secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:15.725056 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:15.723088 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials podName:f39f9edd-f183-4e45-aa47-1eed0e9b5f8c nodeName:}" failed. No retries permitted until 2026-11-04 00:05:16.223060602 +0000 UTC m=+2135.045656759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c") : secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:15.725056 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:15.723640 6208 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-plugins-e2e-extmod-a-3a002b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found Nov 04 00:05:15.725056 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:15.723692 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume podName:f39f9edd-f183-4e45-aa47-1eed0e9b5f8c nodeName:}" failed. No retries permitted until 2026-11-04 00:05:16.223677671 +0000 UTC m=+2135.046273825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found Nov 04 00:05:16.323408 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:16.323378 6208 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a002b/agc-cred-f9c874ea3ea4: secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:16.323408 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:16.323404 6208 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2: secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:16.323852 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:16.323435 6208 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-plugins-e2e-extmod-a-3a002b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found Nov 04 00:05:16.323852 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:16.323462 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials podName:f39f9edd-f183-4e45-aa47-1eed0e9b5f8c nodeName:}" failed. No retries permitted until 2026-11-04 00:05:17.323444434 +0000 UTC m=+2136.146040589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c") : secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:16.323852 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:16.323500 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume podName:f39f9edd-f183-4e45-aa47-1eed0e9b5f8c nodeName:}" failed. No retries permitted until 2026-11-04 00:05:17.323487015 +0000 UTC m=+2136.146083155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found Nov 04 00:05:17.189261 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:17.189221 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:17.342502 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:17.342466 6208 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a002b/agc-cred-f9c874ea3ea4: secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:17.342502 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:17.342504 6208 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2: secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:17.343023 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:17.342485 6208 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-plugins-e2e-extmod-a-3a002b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found Nov 04 00:05:17.343023 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:17.342565 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials podName:f39f9edd-f183-4e45-aa47-1eed0e9b5f8c nodeName:}" failed. No retries permitted until 2026-11-04 00:05:19.342545914 +0000 UTC m=+2138.165142053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c") : secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:17.343023 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:17.342617 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume podName:f39f9edd-f183-4e45-aa47-1eed0e9b5f8c nodeName:}" failed. No retries permitted until 2026-11-04 00:05:19.342605784 +0000 UTC m=+2138.165201929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found Nov 04 00:05:17.918932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:17.918903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:18.025672 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:18.025631 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:18.150179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:18.150138 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:18.942260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:18.942224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:19.375511 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:19.375480 6208 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-plugins-e2e-extmod-a-3a002b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found Nov 04 00:05:19.375691 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:19.375544 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume podName:f39f9edd-f183-4e45-aa47-1eed0e9b5f8c nodeName:}" failed. No retries permitted until 2026-11-04 00:05:23.375530941 +0000 UTC m=+2142.198127094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3a002b" not found Nov 04 00:05:19.375691 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:19.375491 6208 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a002b/agc-cred-f9c874ea3ea4: secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:19.375691 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:19.375589 6208 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2: secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:19.375691 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:19.375637 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials podName:f39f9edd-f183-4e45-aa47-1eed0e9b5f8c nodeName:}" failed. No retries permitted until 2026-11-04 00:05:23.37562357 +0000 UTC m=+2142.198219724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c") : secret "agc-cred-f9c874ea3ea4" not found Nov 04 00:05:21.601273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.601209 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" podUID="f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" containerName="payload-processing" containerID="cri-o://6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725" gracePeriod=25 Nov 04 00:05:21.604723 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.604694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:21.885959 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.885936 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:21.918063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.917632 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-kube-api-access-bpddm\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-kube-api-access-bpddm\") pod \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " Nov 04 00:05:21.918063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.917813 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-routing-overlay-volume\") pod \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " Nov 04 00:05:21.918063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.917963 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials\") pod \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " Nov 04 00:05:21.918317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.918294 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume\") pod \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\" (UID: \"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c\") " Nov 04 00:05:21.920272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.919907 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume" pod "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:05:21.921168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.920845 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-routing-overlay-volume" pod "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:05:21.936576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.936540 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials" pod "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:05:21.941945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:21.941519 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-kube-api-access-bpddm" pod "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" (UID: "f39f9edd-f183-4e45-aa47-1eed0e9b5f8c"). InnerVolumeSpecName "kube-api-access-bpddm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:05:22.024436 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.024394 6208 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-routing-overlay-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:05:22.024436 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.024431 6208 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-provider-credentials\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:05:22.024436 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.024441 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:05:22.024677 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.024450 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpddm\" (UniqueName: \"kubernetes.io/projected/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c-kube-api-access-bpddm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:05:22.672566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.672529 6208 generic.go:356] "Generic (PLEG): container finished" podID="f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" containerID="6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725" exitCode=0 Nov 04 00:05:22.673119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.672602 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" Nov 04 00:05:22.673119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.672608 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" event={"ID":"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c","Type":"ContainerDied","Data":"6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725"} Nov 04 00:05:22.673119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.672681 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2" event={"ID":"f39f9edd-f183-4e45-aa47-1eed0e9b5f8c","Type":"ContainerDied","Data":"a60898e744e2c943741f130e80cd097a288b5bd0910da27bea3119a439ef1bf2"} Nov 04 00:05:22.673119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.672714 6208 scope.go:122] "RemoveContainer" containerID="6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725" Nov 04 00:05:22.685854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.685834 6208 scope.go:122] "RemoveContainer" containerID="6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725" Nov 04 00:05:22.686181 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:22.686159 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725\": container with ID starting with 6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725 not found: ID does not exist" containerID="6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725" Nov 04 00:05:22.686241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.686189 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725"} err="failed to get container status \"6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725\": rpc error: code = NotFound desc = could not find container \"6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725\": container with ID starting with 6262b435835f5b2241f8d01af40c8806e6f4ac507d6cde203147531b97996725 not found: ID does not exist" Nov 04 00:05:22.701704 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.701679 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2"] Nov 04 00:05:22.705490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.705466 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-6d95ntfx2"] Nov 04 00:05:22.966220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:22.966177 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:23.296363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:23.296283 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" path="/var/lib/kubelet/pods/f39f9edd-f183-4e45-aa47-1eed0e9b5f8c/volumes" Nov 04 00:05:23.471629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:23.471596 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:23.600136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:23.599879 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:24.513059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:24.505137 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:24.910818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:24.910699 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:25.341836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.341800 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l"] Nov 04 00:05:25.342787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.342760 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" containerName="payload-processing" Nov 04 00:05:25.342787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.342784 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" containerName="payload-processing" Nov 04 00:05:25.342989 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.342893 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39f9edd-f183-4e45-aa47-1eed0e9b5f8c" containerName="payload-processing" Nov 04 00:05:25.357509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.357479 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.365535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.365509 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l"] Nov 04 00:05:25.366373 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.366311 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c7d218-openshift-default-dockercfg-z5xv6\"" Nov 04 00:05:25.396739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.396702 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rdv\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-kube-api-access-j8rdv\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.396904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.396820 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-certs\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.396904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.396892 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-data\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.397087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.396926 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-token\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.397087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.396976 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.397345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.397325 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-envoy\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.397419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.397397 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istiod-ca-cert\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.397635 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.397598 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-credential-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.397899 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.397880 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-podinfo\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.398007 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.397904 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:25.499311 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499274 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-envoy\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.499490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499355 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istiod-ca-cert\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.499490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499397 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-credential-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.499490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499479 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-podinfo\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.499673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499536 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rdv\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-kube-api-access-j8rdv\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.499673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499603 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-certs\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.499673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499666 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-data\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.499820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499698 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-token\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.499820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499762 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.500084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.499941 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-credential-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.500084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.500021 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-data\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.500216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.500178 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-socket\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.500387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.500359 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-certs\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.500986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.500953 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istiod-ca-cert\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.503417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.503395 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-envoy\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.512201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.512158 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:25.514197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.514167 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rdv\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-kube-api-access-j8rdv\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.550057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.527220 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-podinfo\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.550878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.550805 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-token\") pod \"e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:25.627294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.627205 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:05:25.675055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:25.674250 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:26.124697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:26.124664 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l"] Nov 04 00:05:26.126344 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:05:26.126311 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1457a0f_48fe_45d4_a180_ae1d84f549ba.slice/crio-d71062344d42651046cbf22a03eaf78e3af966ba7da0f30b233e68c84e8acb4b WatchSource:0}: Error finding container d71062344d42651046cbf22a03eaf78e3af966ba7da0f30b233e68c84e8acb4b: Status 404 returned error can't find the container with id d71062344d42651046cbf22a03eaf78e3af966ba7da0f30b233e68c84e8acb4b Nov 04 00:05:26.131905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:26.131862 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:05:26.132010 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:26.131953 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:05:26.132010 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:26.131994 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:05:26.766969 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:26.766938 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:26.783233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:26.783195 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" event={"ID":"b1457a0f-48fe-45d4-a180-ae1d84f549ba","Type":"ContainerStarted","Data":"407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2"} Nov 04 00:05:26.783417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:26.783261 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" event={"ID":"b1457a0f-48fe-45d4-a180-ae1d84f549ba","Type":"ContainerStarted","Data":"d71062344d42651046cbf22a03eaf78e3af966ba7da0f30b233e68c84e8acb4b"} Nov 04 00:05:26.811617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:26.811562 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" podStartSLOduration=1.8115495369999999 podStartE2EDuration="1.811549537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:05:26.807880002 +0000 UTC m=+2145.630476165" watchObservedRunningTime="2026-11-04 00:05:26.811549537 +0000 UTC m=+2145.634145698" Nov 04 00:05:27.308653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:27.308619 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:27.685195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:27.684996 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:27.691566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:27.691538 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:27.790545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:27.790511 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:27.791620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:27.791588 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:05:29.033804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:29.033769 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:29.493378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:29.493340 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:29.605498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:29.605460 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:30.808349 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:30.808312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:30.920147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:30.920110 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:31.018935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:31.018899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:32.492822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:32.492785 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:33.677796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:33.677760 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:33.779491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:33.779345 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:34.799602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:34.799569 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:35.709260 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:05:35.709214 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:05:36.665739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:36.665703 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:37.597818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:37.597778 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:38.579586 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:38.579532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:39.733249 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:39.733191 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:39.875185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:39.875148 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:40.679653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:40.679616 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:40.797717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:40.797680 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:41.541616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:41.541577 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:43.195868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:43.195831 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:44.906083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:44.906011 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:45.979649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:45.979618 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:47.005859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:47.005826 6208 generic.go:356] "Generic (PLEG): container finished" podID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerID="84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271" exitCode=0 Nov 04 00:05:47.006281 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:47.005895 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" event={"ID":"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb","Type":"ContainerDied","Data":"84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271"} Nov 04 00:05:47.006339 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:47.006291 6208 scope.go:122] "RemoveContainer" containerID="84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271" Nov 04 00:05:47.008063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:47.008003 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:05:47.008168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:47.008087 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:05:47.008168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:47.008121 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:05:47.656415 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:47.656332 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:48.015595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:48.015556 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" event={"ID":"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb","Type":"ContainerStarted","Data":"1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25"} Nov 04 00:05:48.881798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:48.881761 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:05:48.882813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:48.882788 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 00:05:48.882889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:48.882851 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 00:05:49.300666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:49.300605 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:49.882107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:49.882059 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 00:05:49.882298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:49.882123 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 00:05:50.543928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:50.543893 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:50.882251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:50.882168 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 00:05:50.882251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:50.882226 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 00:05:51.882535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:51.882497 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 00:05:51.882933 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:51.882571 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 00:05:52.067060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:52.065129 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:05:52.410719 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:52.410690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:52.882557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:52.882515 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 00:05:52.882982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:52.882623 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 00:05:53.644390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:53.644353 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:53.748730 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:53.748692 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:54.290667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:54.290625 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:54.539402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:54.539370 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:05:54.554337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:54.554256 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:05:54.581933 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:54.581899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:55.810289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:55.810245 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:56.138986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:56.138896 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:56.999459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:56.999424 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:57.100271 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:57.100228 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:57.320886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:57.320778 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:57.644085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:57.643958 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:57.805651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:57.805573 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:57.920503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:57.920341 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:58.273407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:58.273369 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:58.463798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:58.463762 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:58.568293 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:58.568189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:05:59.082537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:05:59.082494 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:01.126010 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:01.125971 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:01.849153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:01.849115 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:01.970630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:01.970599 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:02.080191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:02.080156 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:02.994414 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:02.994366 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:04.652003 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:04.651971 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:05.709258 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:06:05.709218 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:06:05.894893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:05.894859 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:07.024671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:07.024633 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:07.734771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:07.734735 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:07.844194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:07.844162 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:09.158378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:09.158343 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:09.376157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:09.376109 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:09.490067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:09.489965 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:09.588532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:09.588488 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:09.848430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:09.848346 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:09.993988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:09.993951 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:10.093092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:10.093056 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:10.284940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:10.284903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:10.868643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:10.868601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:10.974275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:10.974245 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:11.356012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:11.355976 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:11.471361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:11.471328 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:13.043798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:13.043765 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:14.201397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:14.201355 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:15.149688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:15.149647 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:16.703354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.703315 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9"] Nov 04 00:06:16.708155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.708014 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.714811 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.714781 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c7d218-serving-cert\"" Nov 04 00:06:16.721151 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.721123 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9"] Nov 04 00:06:16.769099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.769028 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:16.791691 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.791658 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.791878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.791733 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53bfbe1-15dc-4b22-8121-5e927de1be27-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.791878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.791795 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-metrics-certs\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.791996 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.791899 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbxn7\" (UniqueName: \"kubernetes.io/projected/e53bfbe1-15dc-4b22-8121-5e927de1be27-kube-api-access-kbxn7\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.869752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.869718 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:16.893249 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.893214 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbxn7\" (UniqueName: \"kubernetes.io/projected/e53bfbe1-15dc-4b22-8121-5e927de1be27-kube-api-access-kbxn7\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.893459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.893441 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.893550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.893534 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53bfbe1-15dc-4b22-8121-5e927de1be27-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.893629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.893616 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-metrics-certs\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.894230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.894206 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53bfbe1-15dc-4b22-8121-5e927de1be27-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.895968 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.895949 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-metrics-certs\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.896064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.895990 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.905957 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.905936 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbxn7\" (UniqueName: \"kubernetes.io/projected/e53bfbe1-15dc-4b22-8121-5e927de1be27-kube-api-access-kbxn7\") pod \"maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:16.991984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:16.991914 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:17.027188 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:17.027133 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:17.319319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:17.319289 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9"] Nov 04 00:06:17.320027 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:06:17.319997 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53bfbe1_15dc_4b22_8121_5e927de1be27.slice/crio-851983b749c4f2d96277359a5d1321a3470c42a5bebbf7be307adc80eb31d8c2 WatchSource:0}: Error finding container 851983b749c4f2d96277359a5d1321a3470c42a5bebbf7be307adc80eb31d8c2: Status 404 returned error can't find the container with id 851983b749c4f2d96277359a5d1321a3470c42a5bebbf7be307adc80eb31d8c2 Nov 04 00:06:17.333096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:17.333061 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" event={"ID":"e53bfbe1-15dc-4b22-8121-5e927de1be27","Type":"ContainerStarted","Data":"851983b749c4f2d96277359a5d1321a3470c42a5bebbf7be307adc80eb31d8c2"} Nov 04 00:06:18.163861 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:18.163820 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7"] Nov 04 00:06:18.164295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:18.164081 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" containerID="cri-o://943655b5d5a40214148bac172f5aab6f7a4258ef2a9c574a617bec000909ce52" gracePeriod=30 Nov 04 00:06:18.341849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:18.341816 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" event={"ID":"e53bfbe1-15dc-4b22-8121-5e927de1be27","Type":"ContainerStarted","Data":"4ea8f6d1c5c0dc114f19b652b6c58432ee1a7384749fef75c9913eac205bece3"} Nov 04 00:06:18.342067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:18.342024 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:18.364212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:18.364144 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" podStartSLOduration=1.744506485 podStartE2EDuration="2.364123485s" totalImagesPullingTime="619.617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:06:18.361954428 +0000 UTC m=+2197.184550593" watchObservedRunningTime="2026-11-04 00:06:18.364123485 +0000 UTC m=+2197.186719648" Nov 04 00:06:19.654991 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:19.654945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:20.223287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:20.223253 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd"] Nov 04 00:06:20.223512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:20.223485 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" containerID="cri-o://10e1529d128f8ca4595a5d657a3bd1546cea687458edeb23647ecec84f5d8595" gracePeriod=30 Nov 04 00:06:20.265228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:20.265190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:20.358564 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:20.358531 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:21.950860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:21.950829 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6"] Nov 04 00:06:21.951298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:21.951092 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" containerID="cri-o://5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30" gracePeriod=30 Nov 04 00:06:21.955270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:21.955243 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79"] Nov 04 00:06:21.955444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:21.955423 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" containerID="cri-o://34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0" gracePeriod=30 Nov 04 00:06:22.617113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:22.617072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:23.206503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.206473 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:23.308554 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.308520 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:23.383207 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.383168 6208 generic.go:356] "Generic (PLEG): container finished" podID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerID="943655b5d5a40214148bac172f5aab6f7a4258ef2a9c574a617bec000909ce52" exitCode=0 Nov 04 00:06:23.383363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.383251 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" event={"ID":"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f","Type":"ContainerDied","Data":"943655b5d5a40214148bac172f5aab6f7a4258ef2a9c574a617bec000909ce52"} Nov 04 00:06:23.408652 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.408621 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:23.479323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.479298 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:06:23.543569 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.543533 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-podinfo\") pod \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " Nov 04 00:06:23.543761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.543600 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-socket\") pod \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " Nov 04 00:06:23.543761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.543738 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-data\") pod \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " Nov 04 00:06:23.543901 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.543887 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-kube-api-access-j9v86\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-kube-api-access-j9v86\") pod \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " Nov 04 00:06:23.543963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.543915 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-envoy\") pod \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " Nov 04 00:06:23.544045 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.543976 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istiod-ca-cert\") pod \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " Nov 04 00:06:23.544045 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.544024 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-certs\") pod \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " Nov 04 00:06:23.544183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.544091 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-credential-socket\") pod \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " Nov 04 00:06:23.544183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.544120 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-token\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-token\") pod \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\" (UID: \"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f\") " Nov 04 00:06:23.544183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.544157 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-socket" pod "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" (UID: "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:23.544585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.544556 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istiod-ca-cert" pod "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" (UID: "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:06:23.544691 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.544646 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-data" pod "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" (UID: "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:23.544909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.544885 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-certs" pod "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" (UID: "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:23.545170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.545138 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-credential-socket" pod "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" (UID: "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:23.545453 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.545430 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:23.547092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.545460 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:23.547092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.545477 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:23.547092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.545536 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:23.547092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.545873 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:23.547895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.547866 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-token" pod "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" (UID: "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:06:23.547983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.547955 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-envoy" pod "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" (UID: "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:23.548048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.548012 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-podinfo" pod "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" (UID: "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:06:23.548888 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.548858 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-kube-api-access-j9v86" pod "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" (UID: "ef7eef87-24c5-4fe0-a7dd-1f766c7a906f"). InnerVolumeSpecName "kube-api-access-j9v86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:06:23.580331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.580299 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:23.647517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.647478 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9v86\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-kube-api-access-j9v86\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:23.647517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.647515 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:23.647720 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.647530 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:23.647720 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.647543 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:23.713809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.713779 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:23.833081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:23.823608 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:24.356067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:24.356024 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:06:24.393984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:24.393948 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" event={"ID":"ef7eef87-24c5-4fe0-a7dd-1f766c7a906f","Type":"ContainerDied","Data":"63bef449e2a0325115483f85eba5620dbde38772f0a4ef36f89e789c6d1831eb"} Nov 04 00:06:24.394227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:24.394019 6208 scope.go:122] "RemoveContainer" containerID="943655b5d5a40214148bac172f5aab6f7a4258ef2a9c574a617bec000909ce52" Nov 04 00:06:24.394227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:24.393967 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7" Nov 04 00:06:24.431880 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:24.431849 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7"] Nov 04 00:06:24.481841 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:24.481809 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-wgnn7"] Nov 04 00:06:24.764583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:24.764551 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:25.198916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.198822 6208 patch_prober.go:29] interesting pod/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:06:25.198916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.198895 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:06:25.297277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.297238 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" path="/var/lib/kubelet/pods/ef7eef87-24c5-4fe0-a7dd-1f766c7a906f/volumes" Nov 04 00:06:25.422873 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.422833 6208 generic.go:356] "Generic (PLEG): container finished" podID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerID="10e1529d128f8ca4595a5d657a3bd1546cea687458edeb23647ecec84f5d8595" exitCode=0 Nov 04 00:06:25.423499 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.422899 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" event={"ID":"02ef8451-9b7e-41ab-b0d4-02987d5f97d2","Type":"ContainerDied","Data":"10e1529d128f8ca4595a5d657a3bd1546cea687458edeb23647ecec84f5d8595"} Nov 04 00:06:25.526246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.526219 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 04 00:06:25.593759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.593718 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istiod-ca-cert\") pod \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " Nov 04 00:06:25.593935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.593791 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-data\") pod \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " Nov 04 00:06:25.593935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.593902 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-envoy\") pod \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " Nov 04 00:06:25.594068 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.593960 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-kube-api-access-v9z55\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-kube-api-access-v9z55\") pod \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " Nov 04 00:06:25.594152 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594096 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-credential-socket\") pod \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " Nov 04 00:06:25.594212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594178 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-token\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-token\") pod \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " Nov 04 00:06:25.594272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594209 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-certs\") pod \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " Nov 04 00:06:25.594327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594276 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istiod-ca-cert" pod "02ef8451-9b7e-41ab-b0d4-02987d5f97d2" (UID: "02ef8451-9b7e-41ab-b0d4-02987d5f97d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:06:25.594397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594323 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-podinfo\") pod \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " Nov 04 00:06:25.594503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594449 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-socket\") pod \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\" (UID: \"02ef8451-9b7e-41ab-b0d4-02987d5f97d2\") " Nov 04 00:06:25.594603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594578 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-certs" pod "02ef8451-9b7e-41ab-b0d4-02987d5f97d2" (UID: "02ef8451-9b7e-41ab-b0d4-02987d5f97d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:25.594883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594859 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-socket" pod "02ef8451-9b7e-41ab-b0d4-02987d5f97d2" (UID: "02ef8451-9b7e-41ab-b0d4-02987d5f97d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:25.595105 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594887 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-credential-socket" pod "02ef8451-9b7e-41ab-b0d4-02987d5f97d2" (UID: "02ef8451-9b7e-41ab-b0d4-02987d5f97d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:25.595105 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.594984 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-data" pod "02ef8451-9b7e-41ab-b0d4-02987d5f97d2" (UID: "02ef8451-9b7e-41ab-b0d4-02987d5f97d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:25.595642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.595605 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:25.595794 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.595779 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:25.595908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.595895 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:25.596087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.596026 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:25.596222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.596208 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:25.597790 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.597762 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-kube-api-access-v9z55" pod "02ef8451-9b7e-41ab-b0d4-02987d5f97d2" (UID: "02ef8451-9b7e-41ab-b0d4-02987d5f97d2"). InnerVolumeSpecName "kube-api-access-v9z55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:06:25.598319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.598184 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-token" pod "02ef8451-9b7e-41ab-b0d4-02987d5f97d2" (UID: "02ef8451-9b7e-41ab-b0d4-02987d5f97d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:06:25.598319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.598205 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-podinfo" pod "02ef8451-9b7e-41ab-b0d4-02987d5f97d2" (UID: "02ef8451-9b7e-41ab-b0d4-02987d5f97d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:06:25.598319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.598246 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-envoy" pod "02ef8451-9b7e-41ab-b0d4-02987d5f97d2" (UID: "02ef8451-9b7e-41ab-b0d4-02987d5f97d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:25.697905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.697868 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:25.697905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.697905 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:25.698181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.697920 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:25.698181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:25.697932 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9z55\" (UniqueName: \"kubernetes.io/projected/02ef8451-9b7e-41ab-b0d4-02987d5f97d2-kube-api-access-v9z55\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:26.434581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:26.434547 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" Nov 04 00:06:26.434581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:26.434573 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd" event={"ID":"02ef8451-9b7e-41ab-b0d4-02987d5f97d2","Type":"ContainerDied","Data":"e02cece774860095fd4df25ea5d0d79a28a6472bbc6dc327df44662382af8b3a"} Nov 04 00:06:26.435103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:26.434640 6208 scope.go:122] "RemoveContainer" containerID="10e1529d128f8ca4595a5d657a3bd1546cea687458edeb23647ecec84f5d8595" Nov 04 00:06:26.442800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:26.442655 6208 patch_prober.go:29] interesting pod/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:06:26.442800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:26.442718 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:06:26.460877 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:26.460848 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd"] Nov 04 00:06:26.464622 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:26.464596 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-nhhfd"] Nov 04 00:06:26.598853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:26.598815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:27.253306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.253140 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 04 00:06:27.279715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.279692 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 04 00:06:27.298776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.298742 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" path="/var/lib/kubelet/pods/02ef8451-9b7e-41ab-b0d4-02987d5f97d2/volumes" Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.421702 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-certs\") pod \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.422634 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-certs" pod "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" (UID: "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.422861 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-data\") pod \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.422914 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-envoy\") pod \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423017 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-data\") pod \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423091 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istiod-ca-cert\") pod \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423132 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istiod-ca-cert\") pod \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423281 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-token\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-token\") pod \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423337 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-envoy\") pod \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423381 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-certs\") pod \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423434 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-token\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-token\") pod \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423514 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-credential-socket\") pod \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423600 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-podinfo\") pod \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423660 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-kube-api-access-qmcpt\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-kube-api-access-qmcpt\") pod \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423693 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-kube-api-access-6rktz\") pod \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423726 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-credential-socket\") pod \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423770 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-podinfo\") pod \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423804 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-socket\") pod \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\" (UID: \"c6c279b6-0231-4d45-b0a8-8f05432e22ae\") " Nov 04 00:06:27.424326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.423844 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-socket\") pod \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\" (UID: \"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f\") " Nov 04 00:06:27.425526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.424754 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.425526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.424773 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-certs" pod "c6c279b6-0231-4d45-b0a8-8f05432e22ae" (UID: "c6c279b6-0231-4d45-b0a8-8f05432e22ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.425526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.424948 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istiod-ca-cert" pod "c6c279b6-0231-4d45-b0a8-8f05432e22ae" (UID: "c6c279b6-0231-4d45-b0a8-8f05432e22ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:06:27.425526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.425008 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-socket" pod "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" (UID: "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.425526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.425157 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istiod-ca-cert" pod "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" (UID: "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:06:27.425526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.425213 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-data" pod "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" (UID: "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.425526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.425335 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-data" pod "c6c279b6-0231-4d45-b0a8-8f05432e22ae" (UID: "c6c279b6-0231-4d45-b0a8-8f05432e22ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.425526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.425371 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-credential-socket" pod "c6c279b6-0231-4d45-b0a8-8f05432e22ae" (UID: "c6c279b6-0231-4d45-b0a8-8f05432e22ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.425526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.425381 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-credential-socket" pod "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" (UID: "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.426368 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.426177 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-socket" pod "c6c279b6-0231-4d45-b0a8-8f05432e22ae" (UID: "c6c279b6-0231-4d45-b0a8-8f05432e22ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.427979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.427953 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-envoy" pod "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" (UID: "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.428238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.428143 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-token" pod "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" (UID: "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:06:27.428238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.428196 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-envoy" pod "c6c279b6-0231-4d45-b0a8-8f05432e22ae" (UID: "c6c279b6-0231-4d45-b0a8-8f05432e22ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:06:27.428378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.428350 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-podinfo" pod "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" (UID: "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:06:27.429191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.429162 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-podinfo" pod "c6c279b6-0231-4d45-b0a8-8f05432e22ae" (UID: "c6c279b6-0231-4d45-b0a8-8f05432e22ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:06:27.429282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.429194 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-token" pod "c6c279b6-0231-4d45-b0a8-8f05432e22ae" (UID: "c6c279b6-0231-4d45-b0a8-8f05432e22ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:06:27.429282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.429264 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-kube-api-access-qmcpt" pod "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" (UID: "c6ed126e-cb5d-4fa5-aaf2-1cc63651849f"). InnerVolumeSpecName "kube-api-access-qmcpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:06:27.429643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.429620 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-kube-api-access-6rktz" pod "c6c279b6-0231-4d45-b0a8-8f05432e22ae" (UID: "c6c279b6-0231-4d45-b0a8-8f05432e22ae"). InnerVolumeSpecName "kube-api-access-6rktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:06:27.444731 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.444707 6208 generic.go:356] "Generic (PLEG): container finished" podID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerID="34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0" exitCode=0 Nov 04 00:06:27.445141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.444754 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" Nov 04 00:06:27.445141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.444785 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" event={"ID":"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f","Type":"ContainerDied","Data":"34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0"} Nov 04 00:06:27.445141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.444845 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79" event={"ID":"c6ed126e-cb5d-4fa5-aaf2-1cc63651849f","Type":"ContainerDied","Data":"62150aeccabfda95329c94497fa9ddffd62303b198d36e601e393f2779a1f4e8"} Nov 04 00:06:27.445141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.444870 6208 scope.go:122] "RemoveContainer" containerID="34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0" Nov 04 00:06:27.447640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.447617 6208 generic.go:356] "Generic (PLEG): container finished" podID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerID="5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30" exitCode=0 Nov 04 00:06:27.447798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.447649 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" event={"ID":"c6c279b6-0231-4d45-b0a8-8f05432e22ae","Type":"ContainerDied","Data":"5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30"} Nov 04 00:06:27.447924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.447812 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" event={"ID":"c6c279b6-0231-4d45-b0a8-8f05432e22ae","Type":"ContainerDied","Data":"3dc5bb9a7e8cb814235bf407828ec64ddcc2cade00022e8697550f05851159bd"} Nov 04 00:06:27.447924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.447666 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6" Nov 04 00:06:27.456826 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.456801 6208 scope.go:122] "RemoveContainer" containerID="34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0" Nov 04 00:06:27.457137 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:06:27.457101 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0\": container with ID starting with 34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0 not found: ID does not exist" containerID="34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0" Nov 04 00:06:27.457137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.457124 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0"} err="failed to get container status \"34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0\": rpc error: code = NotFound desc = could not find container \"34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0\": container with ID starting with 34d609da5fc191202aa2b87b09407b56b0a0af523d82e5f2aca8bd9fba0d5dd0 not found: ID does not exist" Nov 04 00:06:27.457262 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.457140 6208 scope.go:122] "RemoveContainer" containerID="5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30" Nov 04 00:06:27.467391 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.467373 6208 scope.go:122] "RemoveContainer" containerID="5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30" Nov 04 00:06:27.468658 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:06:27.468623 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30\": container with ID starting with 5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30 not found: ID does not exist" containerID="5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30" Nov 04 00:06:27.468766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.468662 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30"} err="failed to get container status \"5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30\": rpc error: code = NotFound desc = could not find container \"5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30\": container with ID starting with 5cbeea8870a53ad3c9825e4564c9594a08e98d4f48baf76ac4494d3d6cba3c30 not found: ID does not exist" Nov 04 00:06:27.472669 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.472645 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79"] Nov 04 00:06:27.483891 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.483839 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-zdb79"] Nov 04 00:06:27.525014 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.524980 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6"] Nov 04 00:06:27.525941 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.525918 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.525998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.525948 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.525998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.525962 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.525998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.525970 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.525998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.525981 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.525998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.525990 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.525998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.525998 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526008 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526016 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526025 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526064 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c279b6-0231-4d45-b0a8-8f05432e22ae-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526074 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmcpt\" (UniqueName: \"kubernetes.io/projected/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-kube-api-access-qmcpt\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526090 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rktz\" (UniqueName: \"kubernetes.io/projected/c6c279b6-0231-4d45-b0a8-8f05432e22ae-kube-api-access-6rktz\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526101 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526109 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526118 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c279b6-0231-4d45-b0a8-8f05432e22ae-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.526256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.526126 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:06:27.541253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:27.541228 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-tq7b6"] Nov 04 00:06:28.320925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:28.320843 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:29.297107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:29.297072 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" path="/var/lib/kubelet/pods/c6c279b6-0231-4d45-b0a8-8f05432e22ae/volumes" Nov 04 00:06:29.297616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:29.297601 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" path="/var/lib/kubelet/pods/c6ed126e-cb5d-4fa5-aaf2-1cc63651849f/volumes" Nov 04 00:06:29.881602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:29.881566 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:30.884850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:30.884801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:31.587159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:31.587109 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:34.165107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:34.165074 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:35.685537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:35.685499 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:35.709098 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:06:35.709065 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:06:37.145380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:37.145345 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:37.673875 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:37.673832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:38.200590 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:38.200554 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:38.311326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:38.311284 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:38.411403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:38.411368 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:38.511777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:38.511737 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:40.566894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:40.566856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:40.664581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:40.664549 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:40.847353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:40.847266 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:42.215758 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:42.215721 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:43.762344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:43.762285 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:45.533229 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:45.533167 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:45.636166 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:45.636130 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:45.894603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:45.894504 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:47.742066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:47.741811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:49.637823 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:49.637772 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:51.779615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:51.779576 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:53.980376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:53.980344 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:55.521098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:55.521056 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:55.934546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:55.934447 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:56.941855 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:56.941799 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:57.822090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:57.822019 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": read tcp 10.133.0.2:41208->10.133.0.158:15021: read: connection reset by peer" start-of-body= Nov 04 00:06:57.822280 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:57.822115 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": read tcp 10.133.0.2:41208->10.133.0.158:15021: read: connection reset by peer" Nov 04 00:06:58.746196 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:58.746165 6208 generic.go:356] "Generic (PLEG): container finished" podID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerID="407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2" exitCode=0 Nov 04 00:06:58.746686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:58.746240 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" event={"ID":"b1457a0f-48fe-45d4-a180-ae1d84f549ba","Type":"ContainerDied","Data":"407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2"} Nov 04 00:06:58.746822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:58.746803 6208 scope.go:122] "RemoveContainer" containerID="407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2" Nov 04 00:06:58.748798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:58.748764 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:06:58.748904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:58.748837 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:06:58.748904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:58.748870 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:06:58.764358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:58.764305 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:06:59.780570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:06:59.780523 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" event={"ID":"b1457a0f-48fe-45d4-a180-ae1d84f549ba","Type":"ContainerStarted","Data":"a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754"} Nov 04 00:07:00.498117 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:00.498054 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:00.684728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:00.684683 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:07:00.685020 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:00.684993 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 00:07:00.685118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:00.685065 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 00:07:01.579451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:01.579409 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:01.724252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:01.724220 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:01.879771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:01.879734 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 00:07:01.909510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:01.909441 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 00:07:02.198907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:02.198817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:02.685024 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:02.684981 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 00:07:02.685511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:02.685062 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 00:07:03.685101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:03.685060 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 00:07:03.685516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:03.685136 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 00:07:03.983911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:03.983872 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb"] Nov 04 00:07:03.984489 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:03.984452 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" containerID="cri-o://848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02" gracePeriod=30 Nov 04 00:07:04.684956 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:04.684914 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 00:07:04.685222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:04.684981 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 00:07:04.981930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:04.981895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:05.690136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:05.690105 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:07:05.690648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:05.690574 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:07:05.691457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:05.691438 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:07:05.708646 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:07:05.708609 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:07:06.055267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:06.055228 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc"] Nov 04 00:07:06.055463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:06.055442 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" containerID="cri-o://a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff" gracePeriod=30 Nov 04 00:07:06.152337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:06.152299 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:06.388018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:06.381601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:07.080560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:07.080501 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:07.444694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:07.444608 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:08.924392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:08.924161 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:09.299293 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.299264 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 04 00:07:09.325282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325241 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-socket\") pod \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " Nov 04 00:07:09.325485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325350 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-data\") pod \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " Nov 04 00:07:09.325485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325425 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-envoy\") pod \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " Nov 04 00:07:09.325613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325485 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-credential-socket\") pod \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " Nov 04 00:07:09.325613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325513 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-podinfo\") pod \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " Nov 04 00:07:09.325613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325570 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-token\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-token\") pod \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " Nov 04 00:07:09.325766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325601 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-socket" pod "adbdb3b7-b3c7-4908-8ced-a4185bafa586" (UID: "adbdb3b7-b3c7-4908-8ced-a4185bafa586"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:09.325766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325648 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-kube-api-access-h7b2g\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-kube-api-access-h7b2g\") pod \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " Nov 04 00:07:09.325766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325725 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-certs\") pod \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " Nov 04 00:07:09.325937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325805 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istiod-ca-cert\") pod \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\" (UID: \"adbdb3b7-b3c7-4908-8ced-a4185bafa586\") " Nov 04 00:07:09.325937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.325852 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-credential-socket" pod "adbdb3b7-b3c7-4908-8ced-a4185bafa586" (UID: "adbdb3b7-b3c7-4908-8ced-a4185bafa586"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:09.326258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.326232 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-certs" pod "adbdb3b7-b3c7-4908-8ced-a4185bafa586" (UID: "adbdb3b7-b3c7-4908-8ced-a4185bafa586"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:09.327939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.326517 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-data" pod "adbdb3b7-b3c7-4908-8ced-a4185bafa586" (UID: "adbdb3b7-b3c7-4908-8ced-a4185bafa586"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:09.327939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.326581 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istiod-ca-cert" pod "adbdb3b7-b3c7-4908-8ced-a4185bafa586" (UID: "adbdb3b7-b3c7-4908-8ced-a4185bafa586"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:07:09.327939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.327275 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:09.327939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.327300 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:09.327939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.327316 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:09.327939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.327328 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:09.327939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.327343 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:09.330498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.330463 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-token" pod "adbdb3b7-b3c7-4908-8ced-a4185bafa586" (UID: "adbdb3b7-b3c7-4908-8ced-a4185bafa586"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:07:09.331676 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.331649 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-podinfo" pod "adbdb3b7-b3c7-4908-8ced-a4185bafa586" (UID: "adbdb3b7-b3c7-4908-8ced-a4185bafa586"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:07:09.332585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.332557 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-envoy" pod "adbdb3b7-b3c7-4908-8ced-a4185bafa586" (UID: "adbdb3b7-b3c7-4908-8ced-a4185bafa586"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:09.333479 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.333456 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-kube-api-access-h7b2g" pod "adbdb3b7-b3c7-4908-8ced-a4185bafa586" (UID: "adbdb3b7-b3c7-4908-8ced-a4185bafa586"). InnerVolumeSpecName "kube-api-access-h7b2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:07:09.428735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.428690 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7b2g\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-kube-api-access-h7b2g\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:09.428735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.428725 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:09.428735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.428735 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:09.428735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.428745 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbdb3b7-b3c7-4908-8ced-a4185bafa586-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:09.991530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.991495 6208 generic.go:356] "Generic (PLEG): container finished" podID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerID="848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02" exitCode=0 Nov 04 00:07:09.992022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.991546 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" Nov 04 00:07:09.992022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.991577 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" event={"ID":"adbdb3b7-b3c7-4908-8ced-a4185bafa586","Type":"ContainerDied","Data":"848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02"} Nov 04 00:07:09.992022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.991648 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb" event={"ID":"adbdb3b7-b3c7-4908-8ced-a4185bafa586","Type":"ContainerDied","Data":"1f29a2b776ca276e55a8abe5d7b0a3a87b9f1e75247a665532f5c481f30c7c79"} Nov 04 00:07:09.992022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:09.991681 6208 scope.go:122] "RemoveContainer" containerID="848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02" Nov 04 00:07:10.006961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:10.006939 6208 scope.go:122] "RemoveContainer" containerID="71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020" Nov 04 00:07:10.022369 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:10.022342 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb"] Nov 04 00:07:10.026602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:10.026580 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-92bvb"] Nov 04 00:07:10.045748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:10.045725 6208 scope.go:122] "RemoveContainer" containerID="848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02" Nov 04 00:07:10.046154 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:07:10.046130 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02\": container with ID starting with 848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02 not found: ID does not exist" containerID="848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02" Nov 04 00:07:10.046254 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:10.046168 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02"} err="failed to get container status \"848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02\": rpc error: code = NotFound desc = could not find container \"848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02\": container with ID starting with 848046cca1466957025a1e34960544196837931f60b74cef3bab574f170f5a02 not found: ID does not exist" Nov 04 00:07:10.046254 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:10.046195 6208 scope.go:122] "RemoveContainer" containerID="71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020" Nov 04 00:07:10.046513 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:07:10.046493 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020\": container with ID starting with 71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020 not found: ID does not exist" containerID="71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020" Nov 04 00:07:10.046551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:10.046524 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020"} err="failed to get container status \"71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020\": rpc error: code = NotFound desc = could not find container \"71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020\": container with ID starting with 71f8e7a57eb7575c7e2f8eccb54835067a2a5c626132af31d9b730d4477d7020 not found: ID does not exist" Nov 04 00:07:10.760236 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:10.760195 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:11.302281 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.302247 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" path="/var/lib/kubelet/pods/adbdb3b7-b3c7-4908-8ced-a4185bafa586/volumes" Nov 04 00:07:11.344127 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.344067 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 04 00:07:11.458872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.458835 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-certs\") pod \"586d3517-e490-46c7-a81a-3be5ff076064\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " Nov 04 00:07:11.459082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.458896 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-istio-token\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-istio-token\") pod \"586d3517-e490-46c7-a81a-3be5ff076064\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " Nov 04 00:07:11.459082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.458976 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/586d3517-e490-46c7-a81a-3be5ff076064-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586d3517-e490-46c7-a81a-3be5ff076064-istiod-ca-cert\") pod \"586d3517-e490-46c7-a81a-3be5ff076064\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " Nov 04 00:07:11.459082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459020 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/586d3517-e490-46c7-a81a-3be5ff076064-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586d3517-e490-46c7-a81a-3be5ff076064-istio-podinfo\") pod \"586d3517-e490-46c7-a81a-3be5ff076064\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " Nov 04 00:07:11.459234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459100 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-envoy\") pod \"586d3517-e490-46c7-a81a-3be5ff076064\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " Nov 04 00:07:11.459234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459132 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-credential-socket\") pod \"586d3517-e490-46c7-a81a-3be5ff076064\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " Nov 04 00:07:11.459234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459167 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-certs" pod "586d3517-e490-46c7-a81a-3be5ff076064" (UID: "586d3517-e490-46c7-a81a-3be5ff076064"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:11.459234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459189 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-kube-api-access-j4c8d\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-kube-api-access-j4c8d\") pod \"586d3517-e490-46c7-a81a-3be5ff076064\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " Nov 04 00:07:11.459459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459262 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-socket\") pod \"586d3517-e490-46c7-a81a-3be5ff076064\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " Nov 04 00:07:11.459459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459295 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-data\") pod \"586d3517-e490-46c7-a81a-3be5ff076064\" (UID: \"586d3517-e490-46c7-a81a-3be5ff076064\") " Nov 04 00:07:11.459459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459425 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-credential-socket" pod "586d3517-e490-46c7-a81a-3be5ff076064" (UID: "586d3517-e490-46c7-a81a-3be5ff076064"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:11.459642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459456 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586d3517-e490-46c7-a81a-3be5ff076064-istiod-ca-cert" pod "586d3517-e490-46c7-a81a-3be5ff076064" (UID: "586d3517-e490-46c7-a81a-3be5ff076064"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:07:11.459642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459616 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-socket" pod "586d3517-e490-46c7-a81a-3be5ff076064" (UID: "586d3517-e490-46c7-a81a-3be5ff076064"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:11.459755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459731 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-data" pod "586d3517-e490-46c7-a81a-3be5ff076064" (UID: "586d3517-e490-46c7-a81a-3be5ff076064"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:11.460014 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.459996 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:11.460154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.460015 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:11.460154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.460026 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:11.460154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.460053 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:11.460154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.460063 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586d3517-e490-46c7-a81a-3be5ff076064-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:11.461376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.461349 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/586d3517-e490-46c7-a81a-3be5ff076064-istio-podinfo" pod "586d3517-e490-46c7-a81a-3be5ff076064" (UID: "586d3517-e490-46c7-a81a-3be5ff076064"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:07:11.461497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.461405 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-envoy" pod "586d3517-e490-46c7-a81a-3be5ff076064" (UID: "586d3517-e490-46c7-a81a-3be5ff076064"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:07:11.461698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.461677 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-kube-api-access-j4c8d" pod "586d3517-e490-46c7-a81a-3be5ff076064" (UID: "586d3517-e490-46c7-a81a-3be5ff076064"). InnerVolumeSpecName "kube-api-access-j4c8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:07:11.461790 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.461707 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-istio-token" pod "586d3517-e490-46c7-a81a-3be5ff076064" (UID: "586d3517-e490-46c7-a81a-3be5ff076064"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:07:11.560871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.560838 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:11.560871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.560868 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586d3517-e490-46c7-a81a-3be5ff076064-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:11.560871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.560879 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586d3517-e490-46c7-a81a-3be5ff076064-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:11.561144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.560891 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4c8d\" (UniqueName: \"kubernetes.io/projected/586d3517-e490-46c7-a81a-3be5ff076064-kube-api-access-j4c8d\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:11.710476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:11.710421 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:12.010083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.009984 6208 generic.go:356] "Generic (PLEG): container finished" podID="586d3517-e490-46c7-a81a-3be5ff076064" containerID="a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff" exitCode=0 Nov 04 00:07:12.010083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.010028 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" Nov 04 00:07:12.010083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.010074 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" event={"ID":"586d3517-e490-46c7-a81a-3be5ff076064","Type":"ContainerDied","Data":"a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff"} Nov 04 00:07:12.010334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.010136 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc" event={"ID":"586d3517-e490-46c7-a81a-3be5ff076064","Type":"ContainerDied","Data":"4839382d4f778f23ad54297e26df18df2d9ca4fc32336031b6ae58309d2d901d"} Nov 04 00:07:12.010334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.010167 6208 scope.go:122] "RemoveContainer" containerID="a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff" Nov 04 00:07:12.022320 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.022297 6208 scope.go:122] "RemoveContainer" containerID="a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff" Nov 04 00:07:12.022586 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:07:12.022564 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff\": container with ID starting with a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff not found: ID does not exist" containerID="a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff" Nov 04 00:07:12.022639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.022595 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff"} err="failed to get container status \"a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff\": rpc error: code = NotFound desc = could not find container \"a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff\": container with ID starting with a404cbaed8c93e129d463f08f234095735420a48dd71b01f94efa882bf9081ff not found: ID does not exist" Nov 04 00:07:12.037964 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.037905 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc"] Nov 04 00:07:12.041607 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.041581 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-g9jzc"] Nov 04 00:07:12.748300 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.748267 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:12.944851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:12.944814 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:13.066144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:13.066056 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:13.279924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:13.279890 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:13.296197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:13.296168 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586d3517-e490-46c7-a81a-3be5ff076064" path="/var/lib/kubelet/pods/586d3517-e490-46c7-a81a-3be5ff076064/volumes" Nov 04 00:07:13.385488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:13.385378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:13.584058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:13.581330 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:13.754586 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:13.754540 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:13.880519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:13.880485 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:14.330204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:14.330174 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:15.654647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:15.654605 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:17.569598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:17.569561 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:19.729339 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:19.729242 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:22.355446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:22.355406 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:23.899867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:23.899828 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:24.442177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:24.442142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:24.591289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:24.591247 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:25.007280 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:25.007240 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:26.371672 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:26.371639 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q"] Nov 04 00:07:26.458167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:26.458111 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:26.584723 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:26.584686 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:26.685789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:26.685699 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:27.234286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:27.234249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:28.739731 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:28.739699 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:30.357193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:30.357151 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:31.372578 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.372530 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" podUID="8d2218fc-0055-4894-a3cf-4864b5a316af" containerName="payload-processing" containerID="cri-o://3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d" gracePeriod=25 Nov 04 00:07:31.665207 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.665175 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:07:31.793985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.793935 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-kube-api-access-q2x4r\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-kube-api-access-q2x4r\") pod \"8d2218fc-0055-4894-a3cf-4864b5a316af\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " Nov 04 00:07:31.794208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.794008 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-plugins-config-volume\") pod \"8d2218fc-0055-4894-a3cf-4864b5a316af\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " Nov 04 00:07:31.794208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.794162 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-provider-credentials\") pod \"8d2218fc-0055-4894-a3cf-4864b5a316af\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " Nov 04 00:07:31.794361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.794214 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-routing-overlay-volume\") pod \"8d2218fc-0055-4894-a3cf-4864b5a316af\" (UID: \"8d2218fc-0055-4894-a3cf-4864b5a316af\") " Nov 04 00:07:31.794443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.794406 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-plugins-config-volume" pod "8d2218fc-0055-4894-a3cf-4864b5a316af" (UID: "8d2218fc-0055-4894-a3cf-4864b5a316af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:07:31.794944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.794917 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-routing-overlay-volume" pod "8d2218fc-0055-4894-a3cf-4864b5a316af" (UID: "8d2218fc-0055-4894-a3cf-4864b5a316af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:07:31.795097 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.794972 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:31.796725 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.796698 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-provider-credentials" pod "8d2218fc-0055-4894-a3cf-4864b5a316af" (UID: "8d2218fc-0055-4894-a3cf-4864b5a316af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:07:31.796840 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.796791 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-kube-api-access-q2x4r" pod "8d2218fc-0055-4894-a3cf-4864b5a316af" (UID: "8d2218fc-0055-4894-a3cf-4864b5a316af"). InnerVolumeSpecName "kube-api-access-q2x4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:07:31.896208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.896109 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2x4r\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-kube-api-access-q2x4r\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:31.896208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.896147 6208 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d2218fc-0055-4894-a3cf-4864b5a316af-provider-credentials\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:31.896208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:31.896170 6208 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d2218fc-0055-4894-a3cf-4864b5a316af-routing-overlay-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:07:32.242473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.242438 6208 generic.go:356] "Generic (PLEG): container finished" podID="8d2218fc-0055-4894-a3cf-4864b5a316af" containerID="3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d" exitCode=0 Nov 04 00:07:32.243022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.242504 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" Nov 04 00:07:32.243022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.242520 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" event={"ID":"8d2218fc-0055-4894-a3cf-4864b5a316af","Type":"ContainerDied","Data":"3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d"} Nov 04 00:07:32.243022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.242578 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q" event={"ID":"8d2218fc-0055-4894-a3cf-4864b5a316af","Type":"ContainerDied","Data":"3dcf361ddfca2cffe3f2af2b1416a066a399edac35ffa3815f35b9cefa51f8c3"} Nov 04 00:07:32.243022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.242610 6208 scope.go:122] "RemoveContainer" containerID="3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d" Nov 04 00:07:32.255687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.255632 6208 scope.go:122] "RemoveContainer" containerID="3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d" Nov 04 00:07:32.256018 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:07:32.255989 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d\": container with ID starting with 3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d not found: ID does not exist" containerID="3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d" Nov 04 00:07:32.256170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.256022 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d"} err="failed to get container status \"3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d\": rpc error: code = NotFound desc = could not find container \"3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d\": container with ID starting with 3e2c3aa59218e3f0281f2c8cf247974f4c0e21357c322ddbe2fd9b9f3304f11d not found: ID does not exist" Nov 04 00:07:32.281590 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.281543 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q"] Nov 04 00:07:32.283942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.283909 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-587b8929q"] Nov 04 00:07:32.480141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:32.480105 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:33.330958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:33.330913 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2218fc-0055-4894-a3cf-4864b5a316af" path="/var/lib/kubelet/pods/8d2218fc-0055-4894-a3cf-4864b5a316af/volumes" Nov 04 00:07:34.263394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:34.263345 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:34.691665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:34.691570 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:34.814909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:34.814873 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:34.930346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:34.930311 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:35.463214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:35.463180 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:35.647876 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:35.647842 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:35.707295 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:07:35.707256 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:07:35.764006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:35.763915 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:36.009717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:36.009683 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:36.109118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:36.109020 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:37.736520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:37.736473 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:37.867680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:37.867644 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:39.757306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:39.757270 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:42.042265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:42.042227 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:42.222707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:42.222668 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:42.340234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:42.340141 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:42.653336 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:42.653287 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:42.875276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:42.875238 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:43.258413 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:43.258376 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:43.688712 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:43.688613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:46.000476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:46.000439 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:47.959584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:47.959548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:48.093637 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:48.093603 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:48.202617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:48.202585 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:49.377419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:49.377390 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:51.156279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:51.156242 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:52.577025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:52.576986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:52.712332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:52.712299 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:52.815490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:52.815112 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:54.281788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:54.281753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:55.750858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:55.750823 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:56.864679 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:56.864639 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:07:58.941712 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:07:58.941670 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:00.757143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:00.757107 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:02.032306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:02.032253 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:03.461663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:03.461621 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:05.403910 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:05.403872 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:05.515294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:05.515255 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:05.758886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:05.756977 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:05.786260 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:08:05.786216 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:08:06.046938 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:06.046849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:07.766489 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:07.766444 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:09.354560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:09.354510 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:10.910363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:10.910319 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:12.817026 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:12.816986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:13.543161 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:13.543115 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:13.663894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:13.663854 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:13.771170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:13.771134 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:14.280880 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:14.280842 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:14.383073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:14.383024 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:16.223988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:16.223939 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:16.712761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:16.712724 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:16.812912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:16.812874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:17.158952 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:17.158846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:18.841228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:18.841183 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:20.620517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:20.620481 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:20.721573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:20.721539 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:20.833337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:20.833299 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:20.937922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:20.937821 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:21.119464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:21.119431 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:21.271574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:21.271540 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:21.383494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:21.383458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:21.614966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:21.614873 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:22.236089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:22.236055 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:22.335384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:22.335346 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:22.768673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:22.768636 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:23.218325 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:23.218283 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:23.317253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:23.317217 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:24.032398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:24.032363 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:26.031708 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:26.031674 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:27.973406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:27.973369 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:30.146463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:30.146422 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:32.241314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:32.241274 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:33.669135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:33.669086 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:33.933158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:33.933069 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:35.712726 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:08:35.712689 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:08:35.879402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:35.879370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:37.363008 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:37.362973 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:37.794010 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:37.793970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:39.227179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:39.227142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:41.092578 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:41.092539 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:42.790306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:42.790266 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:44.276077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:44.276021 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:44.479781 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:44.479743 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:45.264100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:45.264059 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:45.929935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:45.929898 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:47.817328 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:47.817293 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:49.449833 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:49.449797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:50.960561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:50.960520 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:53.040099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:53.040060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:55.533258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:55.533208 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:56.753451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:56.753414 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:08:58.356661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:08:58.356624 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:00.050026 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:00.049988 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:01.968540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:01.968505 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:03.969701 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:03.969663 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:05.596638 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:05.596584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:05.714555 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:09:05.714517 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:09:07.521634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:07.521584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:09.775199 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:09.775144 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:11.085320 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:11.085279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:11.188024 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:11.187984 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:11.396217 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:11.396119 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:11.801536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:11.801502 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:11.908363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:11.908324 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:12.076541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:12.076446 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:12.261292 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:12.261256 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:12.372617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:12.372533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:12.608611 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:12.608571 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:13.135895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:13.135864 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:13.236260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:13.236192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:13.741218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:13.741183 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:13.970697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:13.970661 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:14.069299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:14.069203 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:14.526838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:14.526801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:14.937326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:14.937229 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:17.219762 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:17.219722 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:18.341552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:18.341517 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:21.532621 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:21.532579 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:22.981070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:22.980910 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:24.816371 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:24.816318 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:26.724446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:26.724401 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:27.246334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:27.246295 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:27.464059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:27.454637 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:28.227559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:28.227518 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:29.563597 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:29.563565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:29.880114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:29.837434 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:30.173276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:30.173180 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:30.273569 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:30.273535 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:30.621283 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:30.621245 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:30.764721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:30.764686 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:30.886775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:30.886680 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:31.039323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:31.039275 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:31.163919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:31.163118 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:31.333454 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:31.333412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:31.931064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:31.931001 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:32.045457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:32.045423 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:32.150421 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:32.150382 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:32.377893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:32.377723 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:32.496517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:32.496482 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:32.967922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:32.967887 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:33.072949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:33.072879 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:33.597295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:33.597254 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:35.362004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:35.361963 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:35.707178 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:09:35.707139 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:09:37.861749 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:37.861715 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:38.362256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:38.362222 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:40.761105 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:40.761065 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:42.338174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:42.338146 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:09:42.395099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:42.395075 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:09:42.915110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:42.915056 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:44.657164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:44.657126 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:46.172759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:46.172713 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:48.024977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:48.024940 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:49.776027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:49.775986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:51.173086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:51.173050 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:53.282755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:53.282712 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:55.356168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:55.356133 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:58.279286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:58.279245 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:09:59.516322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:09:59.516285 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:01.133801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:01.133768 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:02.613726 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:02.613687 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:05.706770 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:10:05.706728 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:10:05.879484 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:05.879432 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:08.583555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:08.583516 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:10.694109 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:10.694064 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:12.181397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:12.181361 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:13.364230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:13.364193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:14.818582 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:14.818523 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:14.945602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:14.945562 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:16.128766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:16.128733 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:16.253062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:16.238962 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:17.041595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:17.041553 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:18.573459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:18.573420 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:19.894638 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:19.894537 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:23.254646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:23.254612 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:24.412351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:24.412315 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:26.256871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:26.256837 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:27.283832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:27.283795 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:27.385280 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:27.385236 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:29.130916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:29.130882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:29.263099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:29.258736 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:29.361232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:29.361194 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:31.193379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:31.193335 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:32.430278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:32.430241 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf"] Nov 04 00:10:32.430729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:32.430449 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" containerID="cri-o://5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5" gracePeriod=30 Nov 04 00:10:33.728763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:33.728724 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:34.737988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:34.737953 6208 patch_prober.go:29] interesting pod/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:10:34.738408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:34.738008 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:10:35.353295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:35.353258 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:35.710314 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:10:35.710273 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:10:37.059950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.059911 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:37.714923 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.714895 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 04 00:10:37.817717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.817685 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-podinfo\") pod \"769e6f98-1a8e-4778-a9d9-074dce32314f\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " Nov 04 00:10:37.817872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.817725 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-socket\") pod \"769e6f98-1a8e-4778-a9d9-074dce32314f\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " Nov 04 00:10:37.817872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.817834 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-token\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-token\") pod \"769e6f98-1a8e-4778-a9d9-074dce32314f\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " Nov 04 00:10:37.817963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.817946 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-certs\") pod \"769e6f98-1a8e-4778-a9d9-074dce32314f\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " Nov 04 00:10:37.818027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818012 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769e6f98-1a8e-4778-a9d9-074dce32314f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e6f98-1a8e-4778-a9d9-074dce32314f-istiod-ca-cert\") pod \"769e6f98-1a8e-4778-a9d9-074dce32314f\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " Nov 04 00:10:37.818152 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818131 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-socket" pod "769e6f98-1a8e-4778-a9d9-074dce32314f" (UID: "769e6f98-1a8e-4778-a9d9-074dce32314f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:10:37.818152 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818148 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-envoy\") pod \"769e6f98-1a8e-4778-a9d9-074dce32314f\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " Nov 04 00:10:37.818309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818253 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-certs" pod "769e6f98-1a8e-4778-a9d9-074dce32314f" (UID: "769e6f98-1a8e-4778-a9d9-074dce32314f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:10:37.818440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818418 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769e6f98-1a8e-4778-a9d9-074dce32314f-istiod-ca-cert" pod "769e6f98-1a8e-4778-a9d9-074dce32314f" (UID: "769e6f98-1a8e-4778-a9d9-074dce32314f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:10:37.818519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818494 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-credential-socket\") pod \"769e6f98-1a8e-4778-a9d9-074dce32314f\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " Nov 04 00:10:37.818618 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818599 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-data\") pod \"769e6f98-1a8e-4778-a9d9-074dce32314f\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " Nov 04 00:10:37.818734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818647 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-kube-api-access-zq6qr\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-kube-api-access-zq6qr\") pod \"769e6f98-1a8e-4778-a9d9-074dce32314f\" (UID: \"769e6f98-1a8e-4778-a9d9-074dce32314f\") " Nov 04 00:10:37.818893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818803 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-credential-socket" pod "769e6f98-1a8e-4778-a9d9-074dce32314f" (UID: "769e6f98-1a8e-4778-a9d9-074dce32314f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:10:37.819011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.818990 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-data" pod "769e6f98-1a8e-4778-a9d9-074dce32314f" (UID: "769e6f98-1a8e-4778-a9d9-074dce32314f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:10:37.819455 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.819432 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:10:37.819554 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.819459 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:10:37.819554 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.819474 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769e6f98-1a8e-4778-a9d9-074dce32314f-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:10:37.819554 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.819491 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:10:37.819554 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.819506 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:10:37.820028 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.819999 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-podinfo" pod "769e6f98-1a8e-4778-a9d9-074dce32314f" (UID: "769e6f98-1a8e-4778-a9d9-074dce32314f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:10:37.820170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.820149 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-token" pod "769e6f98-1a8e-4778-a9d9-074dce32314f" (UID: "769e6f98-1a8e-4778-a9d9-074dce32314f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:10:37.820423 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.820398 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-envoy" pod "769e6f98-1a8e-4778-a9d9-074dce32314f" (UID: "769e6f98-1a8e-4778-a9d9-074dce32314f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:10:37.820751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.820735 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-kube-api-access-zq6qr" pod "769e6f98-1a8e-4778-a9d9-074dce32314f" (UID: "769e6f98-1a8e-4778-a9d9-074dce32314f"). InnerVolumeSpecName "kube-api-access-zq6qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:10:37.916155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.916121 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:37.921285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.921256 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:10:37.921285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.921288 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:10:37.921446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.921298 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq6qr\" (UniqueName: \"kubernetes.io/projected/769e6f98-1a8e-4778-a9d9-074dce32314f-kube-api-access-zq6qr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:10:37.921446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:37.921309 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769e6f98-1a8e-4778-a9d9-074dce32314f-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:10:38.032897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.032811 6208 generic.go:356] "Generic (PLEG): container finished" podID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerID="5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5" exitCode=0 Nov 04 00:10:38.032897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.032853 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" event={"ID":"769e6f98-1a8e-4778-a9d9-074dce32314f","Type":"ContainerDied","Data":"5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5"} Nov 04 00:10:38.032897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.032880 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" Nov 04 00:10:38.033204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.032899 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf" event={"ID":"769e6f98-1a8e-4778-a9d9-074dce32314f","Type":"ContainerDied","Data":"74aa54aa323c782539e229073b5196c187216ec1cdea5635097ab2447427196b"} Nov 04 00:10:38.033204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.032924 6208 scope.go:122] "RemoveContainer" containerID="5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5" Nov 04 00:10:38.045822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.045803 6208 scope.go:122] "RemoveContainer" containerID="0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84" Nov 04 00:10:38.056230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.056208 6208 scope.go:122] "RemoveContainer" containerID="5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5" Nov 04 00:10:38.056543 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:10:38.056520 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5\": container with ID starting with 5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5 not found: ID does not exist" containerID="5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5" Nov 04 00:10:38.056596 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.056551 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5"} err="failed to get container status \"5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5\": rpc error: code = NotFound desc = could not find container \"5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5\": container with ID starting with 5ab92c1f4651322e44201045b7276d5993b3e073f44f782711baa45b09d245e5 not found: ID does not exist" Nov 04 00:10:38.056596 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.056569 6208 scope.go:122] "RemoveContainer" containerID="0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84" Nov 04 00:10:38.056856 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:10:38.056836 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84\": container with ID starting with 0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84 not found: ID does not exist" containerID="0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84" Nov 04 00:10:38.056949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.056862 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84"} err="failed to get container status \"0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84\": rpc error: code = NotFound desc = could not find container \"0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84\": container with ID starting with 0b7d63301a816bc1eaada841f86e5d797fe39497c69ffbcba82c37c4a1070c84 not found: ID does not exist" Nov 04 00:10:38.059999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.059971 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf"] Nov 04 00:10:38.063482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.063459 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-d6vhf"] Nov 04 00:10:38.568173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:38.568144 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:39.295541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:39.295507 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" path="/var/lib/kubelet/pods/769e6f98-1a8e-4778-a9d9-074dce32314f/volumes" Nov 04 00:10:41.550863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:41.550828 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:41.747061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:41.746987 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:41.985936 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:41.985895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:42.163473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:42.163435 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:42.599057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:42.579496 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:43.142553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:43.142490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:43.286058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:43.271298 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:43.709003 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:43.708961 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:43.849056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:43.835315 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:44.571502 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:44.571471 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:46.252874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:46.252838 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:47.689989 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:47.689954 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:47.801009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:47.800972 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:48.680516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:48.680478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:48.817084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:48.807637 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:48.953924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:48.953823 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:49.055222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:49.055185 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:49.334058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:49.331416 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:51.120315 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:51.120276 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:51.244056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:51.239621 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:51.393634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:51.393544 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:51.555616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:51.555584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:53.269371 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:53.269333 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:55.879764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:55.879723 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:56.482522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:56.482478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:56.591066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:56.591016 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:56.706911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:56.706872 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:57.341679 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:57.341641 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:57.915083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:57.915024 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:58.015822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:58.015784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:58.848587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:58.848552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:58.992576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:58.992543 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:10:59.360693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:10:59.360654 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:00.412545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:00.412505 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:01.757479 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:01.757446 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:02.447247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:02.447214 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:02.580055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:02.575568 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:04.346837 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:04.346802 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:04.445499 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:04.445460 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:04.558670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:04.558624 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:04.868411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:04.868358 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:05.344333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:05.344294 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:05.540904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:05.540867 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:05.759067 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:11:05.750925 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:11:06.678677 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:06.678632 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:07.784355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:07.784308 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:07.884845 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:07.884812 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:08.423451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:08.423398 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:08.840948 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:08.840913 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:10.912849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:10.912813 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:11.186783 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:11.186684 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:11.291048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:11.290996 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:12.678715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:12.678679 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:14.350125 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:14.350081 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:15.368978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:15.368937 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:15.467950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:15.467916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:15.569804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:15.569767 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:15.905640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:15.905604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:16.607957 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:16.607910 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:18.090465 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:18.090425 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:20.009668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:20.009629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:21.339061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:21.339003 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:22.933355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:22.933317 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:23.042733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:23.042700 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:23.720182 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:23.720149 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:23.863546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:23.863491 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:24.001772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:24.001682 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:25.457892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:25.457841 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:27.194366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:27.194325 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:28.820059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:28.816436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:30.661937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:30.661896 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:33.185866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:33.185818 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:35.019643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:35.019599 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:35.708544 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:11:35.708491 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:11:35.794058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:35.790167 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:35.887886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:35.887851 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:36.459384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:36.459347 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:37.170425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:37.170388 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:37.442556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:37.442426 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:38.997146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:38.997105 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:40.896743 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:40.896702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:42.357216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:42.357172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:43.937376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:43.937325 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:44.076606 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:44.076567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:44.179719 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:44.179678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:44.282566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:44.282533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:46.015648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:46.015611 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:47.675060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:47.675011 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:47.989469 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:47.989428 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:49.867868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:49.867771 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:50.762265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:50.762213 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:50.868000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:50.867963 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:51.543405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.543367 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp"] Nov 04 00:11:51.544018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544005 6208 cpu_manager.go:414] "containerMap: removing container" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" Nov 04 00:11:51.544093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544020 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" Nov 04 00:11:51.544093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544050 6208 cpu_manager.go:414] "containerMap: removing container" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" Nov 04 00:11:51.544093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544058 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" Nov 04 00:11:51.544093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544073 6208 cpu_manager.go:414] "containerMap: removing container" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" Nov 04 00:11:51.544093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544080 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544096 6208 cpu_manager.go:414] "containerMap: removing container" podUID="8d2218fc-0055-4894-a3cf-4864b5a316af" containerName="payload-processing" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544103 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2218fc-0055-4894-a3cf-4864b5a316af" containerName="payload-processing" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544126 6208 cpu_manager.go:414] "containerMap: removing container" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544131 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544141 6208 cpu_manager.go:414] "containerMap: removing container" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544147 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544163 6208 cpu_manager.go:414] "containerMap: removing container" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544169 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544180 6208 cpu_manager.go:414] "containerMap: removing container" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544185 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544193 6208 cpu_manager.go:414] "containerMap: removing container" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" Nov 04 00:11:51.544248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544197 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544325 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c279b6-0231-4d45-b0a8-8f05432e22ae" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544334 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7eef87-24c5-4fe0-a7dd-1f766c7a906f" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544341 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544349 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbdb3b7-b3c7-4908-8ced-a4185bafa586" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544354 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ef8451-9b7e-41ab-b0d4-02987d5f97d2" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544362 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544369 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ed126e-cb5d-4fa5-aaf2-1cc63651849f" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544375 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544382 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="586d3517-e490-46c7-a81a-3be5ff076064" containerName="istio-proxy" Nov 04 00:11:51.544616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.544387 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2218fc-0055-4894-a3cf-4864b5a316af" containerName="payload-processing" Nov 04 00:11:51.547630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.547610 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.563539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.563511 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp"] Nov 04 00:11:51.593204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.593158 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0530c0-024b-438e-8cd4-e530338b9744-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.593543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.593503 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.593671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.593614 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.593671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.593648 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggx9\" (UniqueName: \"kubernetes.io/projected/eb0530c0-024b-438e-8cd4-e530338b9744-kube-api-access-6ggx9\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.593806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.593742 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0530c0-024b-438e-8cd4-e530338b9744-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.593806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.593778 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.593927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.593845 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0530c0-024b-438e-8cd4-e530338b9744-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.593927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.593915 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.594090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.593994 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.695282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695249 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0530c0-024b-438e-8cd4-e530338b9744-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.695282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695289 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.695631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695336 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0530c0-024b-438e-8cd4-e530338b9744-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.695631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695392 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.695631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695501 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.695631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695581 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0530c0-024b-438e-8cd4-e530338b9744-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.695905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695639 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.695905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695717 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.695905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695738 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggx9\" (UniqueName: \"kubernetes.io/projected/eb0530c0-024b-438e-8cd4-e530338b9744-kube-api-access-6ggx9\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.696124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.695916 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.696189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.696162 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.696643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.696596 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.696751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.696720 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb0530c0-024b-438e-8cd4-e530338b9744-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.696853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.696814 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.698392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.698365 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb0530c0-024b-438e-8cd4-e530338b9744-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.698481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.698413 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb0530c0-024b-438e-8cd4-e530338b9744-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.708481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.708458 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb0530c0-024b-438e-8cd4-e530338b9744-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.708635 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.708583 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggx9\" (UniqueName: \"kubernetes.io/projected/eb0530c0-024b-438e-8cd4-e530338b9744-kube-api-access-6ggx9\") pod \"maas-default-gateway-openshift-default-778b7b5c47-ldsgp\" (UID: \"eb0530c0-024b-438e-8cd4-e530338b9744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:51.763294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.763259 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:51.862076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.861958 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:51.872151 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:51.872123 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:52.170832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:52.170585 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:52.401835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:52.401804 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp"] Nov 04 00:11:52.402513 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:11:52.402486 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0530c0_024b_438e_8cd4_e530338b9744.slice/crio-11d2e94b314bc0be6df991b28b511d87c151a00c2e935b3610c5b5f6eb9a995d WatchSource:0}: Error finding container 11d2e94b314bc0be6df991b28b511d87c151a00c2e935b3610c5b5f6eb9a995d: Status 404 returned error can't find the container with id 11d2e94b314bc0be6df991b28b511d87c151a00c2e935b3610c5b5f6eb9a995d Nov 04 00:11:52.405360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:52.405322 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:11:52.405464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:52.405405 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:11:52.405531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:52.405468 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:11:53.201179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:53.201146 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:53.328198 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:53.326251 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:53.409058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:53.408320 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" event={"ID":"eb0530c0-024b-438e-8cd4-e530338b9744","Type":"ContainerStarted","Data":"297dc47b4e37f22393ad7e36f5c65ac90d88a6d3e04b9d3976b5f2b1f91840e9"} Nov 04 00:11:53.409058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:53.408408 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" event={"ID":"eb0530c0-024b-438e-8cd4-e530338b9744","Type":"ContainerStarted","Data":"11d2e94b314bc0be6df991b28b511d87c151a00c2e935b3610c5b5f6eb9a995d"} Nov 04 00:11:53.416863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:53.416750 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podStartSLOduration=2.416729598 podStartE2EDuration="2.416729598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:11:53.412813169 +0000 UTC m=+2532.235409334" watchObservedRunningTime="2026-11-04 00:11:53.416729598 +0000 UTC m=+2532.239325763" Nov 04 00:11:53.878222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:53.878185 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:11:53.879218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:53.879182 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 00:11:53.879386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:53.879248 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 00:11:54.274057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:54.272208 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:54.878189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:54.878147 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 00:11:54.878416 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:54.878230 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 00:11:55.877059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:55.876196 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 00:11:55.877059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:55.876263 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 00:11:56.308144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:56.308091 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:56.878191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:56.878139 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 00:11:56.878721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:56.878232 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 00:11:57.876158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:57.874465 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 00:11:57.876158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:57.874544 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 00:11:58.028001 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:58.027970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:58.873095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:58.873007 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 00:11:58.873288 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:58.873128 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 00:11:59.652590 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:59.652552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:11:59.872878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:59.872838 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 00:11:59.873085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:11:59.872915 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 00:12:01.365103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:01.365065 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:01.502061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:01.498729 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:12:01.502061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:01.500779 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:12:01.516069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:01.514744 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" Nov 04 00:12:02.623505 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:02.623466 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:03.490133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:03.490073 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:03.619186 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:03.619143 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:04.538286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:04.538255 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:04.826821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:04.826721 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:05.036730 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:05.036633 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:05.262019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:05.261975 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:05.492674 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:05.492633 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:05.720688 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:12:05.720646 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:12:06.221572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:06.221520 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:06.351348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:06.351314 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:06.479204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:06.479100 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:06.831162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:06.831083 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p"] Nov 04 00:12:06.832229 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:06.832202 6208 cpu_manager.go:414] "containerMap: removing container" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" Nov 04 00:12:06.832335 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:06.832232 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769e6f98-1a8e-4778-a9d9-074dce32314f" containerName="istio-proxy" Nov 04 00:12:06.837962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:06.837937 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p"] Nov 04 00:12:06.838274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:06.838254 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:06.918467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:06.918197 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:07.020780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.020746 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.020950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.020886 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2726d5-efb3-4832-977b-e2315e48a63c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.020950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.020925 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.021067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.021017 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.021240 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.021224 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2726d5-efb3-4832-977b-e2315e48a63c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.021323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.021298 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.021390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.021370 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2726d5-efb3-4832-977b-e2315e48a63c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.021497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.021483 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.021562 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.021513 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksk7\" (UniqueName: \"kubernetes.io/projected/de2726d5-efb3-4832-977b-e2315e48a63c-kube-api-access-6ksk7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.122528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122442 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.122528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122480 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2726d5-efb3-4832-977b-e2315e48a63c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.122528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122508 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.122791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122540 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2726d5-efb3-4832-977b-e2315e48a63c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.122791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122608 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.122791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122635 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksk7\" (UniqueName: \"kubernetes.io/projected/de2726d5-efb3-4832-977b-e2315e48a63c-kube-api-access-6ksk7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.122791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122744 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.123015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122859 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2726d5-efb3-4832-977b-e2315e48a63c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.123015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122897 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.123015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122911 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.123015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.122949 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.123220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.123196 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.123220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.123214 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.123421 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.123393 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2726d5-efb3-4832-977b-e2315e48a63c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.124982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.124957 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2726d5-efb3-4832-977b-e2315e48a63c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.125387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.125370 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2726d5-efb3-4832-977b-e2315e48a63c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.135282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.135253 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2726d5-efb3-4832-977b-e2315e48a63c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.138381 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.138358 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksk7\" (UniqueName: \"kubernetes.io/projected/de2726d5-efb3-4832-977b-e2315e48a63c-kube-api-access-6ksk7\") pod \"maas-default-gateway-openshift-default-778b7b5c47-5hv2p\" (UID: \"de2726d5-efb3-4832-977b-e2315e48a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.242059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.242009 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:07.389825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.389736 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:07.490111 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.490068 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:07.586110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.576373 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p"] Nov 04 00:12:07.589750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.587603 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:12:07.589750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.587712 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:12:07.589750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.587759 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:12:07.772651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.772610 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:07.873594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.873541 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:07.924088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.923979 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" event={"ID":"de2726d5-efb3-4832-977b-e2315e48a63c","Type":"ContainerStarted","Data":"9da185437cb33ad324ca2bd2a4c1d58182f61689ce40ca66596d7372e0779b3c"} Nov 04 00:12:07.924272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.924121 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" event={"ID":"de2726d5-efb3-4832-977b-e2315e48a63c","Type":"ContainerStarted","Data":"ac997d426ef605c56cdebc9cb2c1c92c891dda8b955a72ec81aea0bb7ba658a0"} Nov 04 00:12:07.952919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:07.952860 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podStartSLOduration=1.952842295 podStartE2EDuration="1.952842295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:12:07.950254237 +0000 UTC m=+2546.772850398" watchObservedRunningTime="2026-11-04 00:12:07.952842295 +0000 UTC m=+2546.775438454" Nov 04 00:12:08.111054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:08.089895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:08.238069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:08.237671 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:08.249065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:08.246723 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:08.249065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:08.248860 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 00:12:08.249065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:08.248912 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 00:12:08.372505 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:08.372473 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:08.676673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:08.676580 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:09.242443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:09.242404 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:09.242950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:09.242695 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 00:12:09.242950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:09.242747 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 00:12:09.348688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:09.348650 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:10.091498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:10.091463 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:10.239583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:10.239547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:10.242587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:10.242560 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 00:12:10.242913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:10.242616 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 00:12:10.450118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:10.427962 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:11.243596 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:11.243556 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 00:12:11.244106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:11.243631 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 00:12:12.198529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:12.198481 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:12.243415 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:12.243381 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 00:12:12.243572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:12.243465 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 00:12:12.841761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:12.841723 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:12.944866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:12.944832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:13.052446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:13.052410 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:13.243341 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:13.243303 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 00:12:13.243553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:13.243361 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 00:12:14.186721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:14.186681 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:14.245721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:14.245676 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 00:12:14.246508 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:14.245990 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 00:12:15.242866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:15.242825 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 00:12:15.243401 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:15.242893 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 00:12:15.906202 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:15.906161 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:17.152024 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:17.151985 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:17.153287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:17.153263 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:17.153928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:17.153904 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" Nov 04 00:12:17.843394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:17.843345 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:19.540513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:19.540468 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:20.397096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:20.397064 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:20.547114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:20.547074 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:20.792405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:20.792366 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:20.948958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:20.948901 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:21.131948 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:21.131860 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:21.251055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:21.250264 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:21.856890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:21.856855 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:21.973691 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:21.973653 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:22.953927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:22.953886 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:23.407612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:23.391737 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:23.579106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:23.544317 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:23.914808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:23.914770 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:25.985058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:25.984631 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:26.730057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:26.722405 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:28.950992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:28.950949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:31.071307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:31.071265 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:33.144447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:33.144412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:34.988634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:34.988601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:35.705760 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:12:35.705592 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:12:36.202550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:36.202458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:37.454780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:37.454744 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:37.661187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:37.661149 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:39.514526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:39.514472 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:42.552329 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:42.552280 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:43.762515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:43.762475 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:45.232156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:45.232119 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:47.154641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:47.154603 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:48.049805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:48.049751 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:48.320091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:48.319958 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:49.757585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:49.757551 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:51.880065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:51.874445 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:54.710057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:54.709072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:55.153531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:55.153443 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:56.258066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:56.258001 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:57.578654 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:57.578614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:58.711802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:58.711764 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:12:59.791981 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:12:59.791947 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:01.050912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:01.050857 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:02.428620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:02.428585 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:03.925283 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:03.925247 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:05.107878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:05.107840 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:05.706212 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:13:05.706173 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:13:06.364646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:06.364610 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:07.678866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:07.678822 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:09.409180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:09.409145 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:11.014272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:11.014231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:12.483534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:12.483489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:15.268869 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:15.268831 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:16.241257 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:16.241212 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:17.316056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:17.315173 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:19.141481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:19.141441 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:20.026180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:20.026089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:20.945904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:20.945869 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:22.466413 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:22.466362 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:23.556798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:23.556758 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:24.365314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:24.365281 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:25.390594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:25.390542 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:26.045010 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:26.044973 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:26.391711 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:26.391619 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:27.353191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:27.353150 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:28.390089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:28.390010 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:29.173100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:29.173064 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:29.286054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:29.285109 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:29.568529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:29.568498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:30.696143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:30.696090 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:31.336506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:31.336465 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:31.436457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:31.436417 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:32.126698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:32.126658 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:32.397894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:32.397766 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:32.507573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:32.507533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:33.458311 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:33.458272 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:34.571916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:34.571858 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:35.706994 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:13:35.706959 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:13:35.726343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:35.726309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:36.650617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:36.650571 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:37.509447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:37.509404 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:37.683803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:37.683771 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:38.946223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:38.946191 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:39.814344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:39.814312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:40.855082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:40.855049 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:41.628890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:41.628851 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:42.444157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:42.444097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:43.437901 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:43.437862 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:43.538369 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:43.538328 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:44.566004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:44.565958 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:44.824219 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:44.824124 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:44.938057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:44.936109 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:45.748286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:45.748247 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:46.656715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:46.656664 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:47.434603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:47.434562 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:48.114250 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:48.114201 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:48.259417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:48.259383 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:49.246626 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:49.246590 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:50.127058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:50.126588 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:51.487060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:51.486058 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:53.140601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:53.140451 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:53.608848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:53.608809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:53.708327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:53.708290 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:54.138317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:54.138278 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:54.473108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:54.469950 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:55.350760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:55.350718 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:56.228697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:56.228663 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:56.351012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:56.350975 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:56.530747 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:56.530640 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:56.796270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:56.796176 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:57.122811 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:57.122705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:57.343394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:57.343356 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:57.443204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:57.443106 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:58.356207 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:58.356170 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:58.686795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:58.686614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:58.858057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:58.842449 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:13:59.721763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:13:59.721728 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:00.454319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:00.454288 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:01.225587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:01.225553 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:02.135732 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:02.135696 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:02.860598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:02.860554 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:03.301521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:03.301489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:03.401693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:03.401660 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:03.694874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:03.694771 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:04.446239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:04.446204 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:05.247559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:05.247506 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:05.706660 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:14:05.706625 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:14:05.959872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:05.959775 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:06.692722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:06.692677 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:07.658963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:07.658926 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:08.634545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:08.634510 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:09.137588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:09.137543 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:09.255054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:09.246609 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:09.542143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:09.542098 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:10.317169 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:10.317130 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:11.067964 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:11.067929 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:11.856820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:11.856786 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:12.620071 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:12.620026 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:13.384435 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:13.384396 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:14.155949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:14.155911 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:15.019749 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.019708 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:15.355291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.355179 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:15.463873 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.463838 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:15.964900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.964862 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7"] Nov 04 00:14:15.970069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.970015 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:15.974358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.974326 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1b5fca\"" Nov 04 00:14:15.975177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.975136 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"routing-overlay\"" Nov 04 00:14:15.975177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.975154 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"openshift-service-ca.crt\"" Nov 04 00:14:15.975383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.975111 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"kube-root-ca.crt\"" Nov 04 00:14:15.975383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.975193 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-559c901bbcfd\"" Nov 04 00:14:15.976307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.975961 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"agc-cred-2df8488e854b\"" Nov 04 00:14:15.976307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.976081 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5fca\"/\"payload-processing-external-model-e2e--0bada0ae-dockercfg-d6htd\"" Nov 04 00:14:15.979309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:15.979281 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7"] Nov 04 00:14:16.050970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.050931 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.051387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.051129 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.051387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.051279 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.051387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.051325 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjnx\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-kube-api-access-8xjnx\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.152716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.152676 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.152936 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.152855 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.153010 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.152977 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.153092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.153014 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjnx\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-kube-api-access-8xjnx\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.153388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.153363 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.153695 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.153671 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.155805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.155781 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.165458 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.165429 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjnx\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-kube-api-access-8xjnx\") pod \"payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.199831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.199792 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8"] Nov 04 00:14:16.204692 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.204675 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.211696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.211317 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"agc-cred-14b358f9ee72\"" Nov 04 00:14:16.211696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.211371 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"routing-overlay\"" Nov 04 00:14:16.211696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.211506 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-e2e--7c418871-dockercfg-7sffv\"" Nov 04 00:14:16.211696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.211528 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"kube-root-ca.crt\"" Nov 04 00:14:16.211696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.211564 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"openshift-service-ca.crt\"" Nov 04 00:14:16.211696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.211684 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3a002b\"" Nov 04 00:14:16.212141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.211983 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a002b\"/\"agc-cred-4ebb40243897\"" Nov 04 00:14:16.212849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.212825 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8"] Nov 04 00:14:16.288738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.288699 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:16.356832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.356799 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.356966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.356903 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4msc\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-kube-api-access-k4msc\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.356966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.356931 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.357225 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.357124 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.439162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.439128 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:16.458605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.458567 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.458757 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.458738 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.458836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.458809 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4msc\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-kube-api-access-k4msc\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.458901 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.458845 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.459259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.459235 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.459540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.459521 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.461273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.461251 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.503536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.502699 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4msc\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-kube-api-access-k4msc\") pod \"payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.541209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.529122 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:16.584448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.584418 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7"] Nov 04 00:14:16.585695 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:14:16.585667 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc4333e_ca3e_40be_931c_1068b60d2915.slice/crio-7573422b76995cc6782472e5159e9d311d875642340618dce22d3645dae4ed82 WatchSource:0}: Error finding container 7573422b76995cc6782472e5159e9d311d875642340618dce22d3645dae4ed82: Status 404 returned error can't find the container with id 7573422b76995cc6782472e5159e9d311d875642340618dce22d3645dae4ed82 Nov 04 00:14:16.853661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:16.853629 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8"] Nov 04 00:14:17.389454 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:17.389405 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:17.390756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:17.390725 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" event={"ID":"bcc4333e-ca3e-40be-931c-1068b60d2915","Type":"ContainerStarted","Data":"608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa"} Nov 04 00:14:17.390909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:17.390789 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" event={"ID":"bcc4333e-ca3e-40be-931c-1068b60d2915","Type":"ContainerStarted","Data":"7573422b76995cc6782472e5159e9d311d875642340618dce22d3645dae4ed82"} Nov 04 00:14:17.390909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:17.390868 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:17.392537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:17.392511 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" event={"ID":"f229cbce-2905-47a2-af7d-33750f50ee81","Type":"ContainerStarted","Data":"64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851"} Nov 04 00:14:17.392673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:17.392560 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" event={"ID":"f229cbce-2905-47a2-af7d-33750f50ee81","Type":"ContainerStarted","Data":"e29829a9984336ef492d775c3b06aac9c87cc9f028605c29964b2081dcb515e2"} Nov 04 00:14:17.392673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:17.392604 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:17.411697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:17.411652 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" podStartSLOduration=2.411638474 podStartE2EDuration="2.411638474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:14:17.409657196 +0000 UTC m=+2676.232253366" watchObservedRunningTime="2026-11-04 00:14:17.411638474 +0000 UTC m=+2676.234234657" Nov 04 00:14:17.471058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:17.435027 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" podStartSLOduration=1.435006777 podStartE2EDuration="1.435006777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:14:17.431747061 +0000 UTC m=+2676.254343247" watchObservedRunningTime="2026-11-04 00:14:17.435006777 +0000 UTC m=+2676.257602940" Nov 04 00:14:18.224917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:18.224872 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:19.014900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:19.014863 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:20.194115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:20.194072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:20.945633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:20.945596 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:21.740073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:21.740023 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:22.466246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:22.466213 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:23.258699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:23.258659 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:24.055524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:24.055485 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:24.950954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:24.950919 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:25.976520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:25.976486 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:26.128617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:26.128581 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:26.909351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:26.909305 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:27.721774 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:27.721738 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:28.402800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:28.402772 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:14:28.403011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:28.402836 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:14:28.539086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:28.539006 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:29.247364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:29.247327 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:30.133335 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:30.133295 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:30.533805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:30.533769 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:30.633610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:30.633571 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:30.847162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:30.847055 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:31.688680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:31.688631 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:32.614335 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:32.614292 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:33.355496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:33.355462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:34.293491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:34.293444 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:35.133590 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:35.133552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:35.709530 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:14:35.709495 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:14:35.831577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:35.831543 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:36.723274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:36.723234 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:37.533440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:37.533402 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:37.635438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:37.635398 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:38.245230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:38.245193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:38.952588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:38.952555 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:39.729179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:39.729143 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:40.519957 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:40.519924 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:41.226534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:41.226488 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:42.036881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:42.036844 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:42.391781 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:42.391706 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:14:42.450005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:42.449976 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:14:42.823082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:42.823046 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:43.558200 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:43.558163 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:44.454898 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:44.454849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:45.459016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:45.458938 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:47.036742 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:47.036671 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:47.551179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:47.551124 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:14:47.551400 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:47.551199 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:14:48.196214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:48.196166 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:14:48.196746 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:48.196242 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:14:48.203732 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:48.203636 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:48.402675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:48.402642 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:49.326103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:49.326066 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:50.130080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:50.130016 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:50.873977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:50.873934 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:51.887894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:51.887848 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:52.740284 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:52.740249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:53.634592 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:53.634557 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:54.528538 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:54.528498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:55.441423 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:55.441377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:56.195270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:56.195228 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:56.923268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:56.923232 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:57.835696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:57.835660 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:58.521100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:58.521063 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:14:59.249159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:14:59.249125 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:00.198806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.198763 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8"] Nov 04 00:15:00.205185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.205151 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8"] Nov 04 00:15:00.205628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.205591 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:00.224535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.224506 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8"] Nov 04 00:15:00.224887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.224868 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:00.231738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.231708 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8"] Nov 04 00:15:00.238888 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.238020 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf"] Nov 04 00:15:00.254310 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.254262 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\"" Nov 04 00:15:00.255044 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.254988 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fzb4\" (UniqueName: \"kubernetes.io/projected/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-kube-api-access-8fzb4\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8\" (UID: \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:00.255338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.255313 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8\" (UID: \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:00.266338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.260279 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:00.266521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.266360 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd"] Nov 04 00:15:00.272117 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.272089 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:00.287249 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.287205 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf"] Nov 04 00:15:00.293438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.293411 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd"] Nov 04 00:15:00.299419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.299387 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584"] Nov 04 00:15:00.305249 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.305222 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr"] Nov 04 00:15:00.305434 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.305348 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:00.310663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.310638 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:00.311758 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.311721 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584"] Nov 04 00:15:00.317209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.317185 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr"] Nov 04 00:15:00.326393 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.326343 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.357492 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8\" (UID: \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.357594 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33426b4e-740d-4fee-abf5-09793b47bdb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584\" (UID: \"33426b4e-740d-4fee-abf5-09793b47bdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.358690 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8\" (UID: \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.359121 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44553b66-1042-4dd2-95cb-76d526b887a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf\" (UID: \"44553b66-1042-4dd2-95cb-76d526b887a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.359324 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsnm\" (UniqueName: \"kubernetes.io/projected/85ef5cef-d3ab-4f83-861f-b64a421a90c0-kube-api-access-dhsnm\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd\" (UID: \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.359523 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfld6\" (UniqueName: \"kubernetes.io/projected/33426b4e-740d-4fee-abf5-09793b47bdb9-kube-api-access-rfld6\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584\" (UID: \"33426b4e-740d-4fee-abf5-09793b47bdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.359751 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ef5cef-d3ab-4f83-861f-b64a421a90c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd\" (UID: \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.359799 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eba2c2-e54d-4443-b017-e5f3e10b1742-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8\" (UID: \"96eba2c2-e54d-4443-b017-e5f3e10b1742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.359888 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztp2k\" (UniqueName: \"kubernetes.io/projected/96eba2c2-e54d-4443-b017-e5f3e10b1742-kube-api-access-ztp2k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8\" (UID: \"96eba2c2-e54d-4443-b017-e5f3e10b1742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.359948 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8684\" (UniqueName: \"kubernetes.io/projected/44553b66-1042-4dd2-95cb-76d526b887a1-kube-api-access-k8684\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf\" (UID: \"44553b66-1042-4dd2-95cb-76d526b887a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.359990 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s8nh\" (UniqueName: \"kubernetes.io/projected/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-kube-api-access-6s8nh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr\" (UID: \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.360099 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr\" (UID: \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:00.378059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.360144 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fzb4\" (UniqueName: \"kubernetes.io/projected/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-kube-api-access-8fzb4\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8\" (UID: \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:00.425147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.414760 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fzb4\" (UniqueName: \"kubernetes.io/projected/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-kube-api-access-8fzb4\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8\" (UID: \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:00.462074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462009 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsnm\" (UniqueName: \"kubernetes.io/projected/85ef5cef-d3ab-4f83-861f-b64a421a90c0-kube-api-access-dhsnm\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd\" (UID: \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:00.462304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462101 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfld6\" (UniqueName: \"kubernetes.io/projected/33426b4e-740d-4fee-abf5-09793b47bdb9-kube-api-access-rfld6\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584\" (UID: \"33426b4e-740d-4fee-abf5-09793b47bdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:00.462304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462157 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ef5cef-d3ab-4f83-861f-b64a421a90c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd\" (UID: \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:00.462304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462195 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eba2c2-e54d-4443-b017-e5f3e10b1742-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8\" (UID: \"96eba2c2-e54d-4443-b017-e5f3e10b1742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:00.462304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462250 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztp2k\" (UniqueName: \"kubernetes.io/projected/96eba2c2-e54d-4443-b017-e5f3e10b1742-kube-api-access-ztp2k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8\" (UID: \"96eba2c2-e54d-4443-b017-e5f3e10b1742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:00.462304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462294 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8684\" (UniqueName: \"kubernetes.io/projected/44553b66-1042-4dd2-95cb-76d526b887a1-kube-api-access-k8684\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf\" (UID: \"44553b66-1042-4dd2-95cb-76d526b887a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:00.462616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462333 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s8nh\" (UniqueName: \"kubernetes.io/projected/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-kube-api-access-6s8nh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr\" (UID: \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:00.462616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462407 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr\" (UID: \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:00.462616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462496 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33426b4e-740d-4fee-abf5-09793b47bdb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584\" (UID: \"33426b4e-740d-4fee-abf5-09793b47bdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:00.462616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.462543 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44553b66-1042-4dd2-95cb-76d526b887a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf\" (UID: \"44553b66-1042-4dd2-95cb-76d526b887a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:00.463274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.463058 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ef5cef-d3ab-4f83-861f-b64a421a90c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd\" (UID: \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:00.463274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.463171 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eba2c2-e54d-4443-b017-e5f3e10b1742-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8\" (UID: \"96eba2c2-e54d-4443-b017-e5f3e10b1742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:00.463447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.463396 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr\" (UID: \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:00.463447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.463410 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33426b4e-740d-4fee-abf5-09793b47bdb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584\" (UID: \"33426b4e-740d-4fee-abf5-09793b47bdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:00.463447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.463396 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44553b66-1042-4dd2-95cb-76d526b887a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf\" (UID: \"44553b66-1042-4dd2-95cb-76d526b887a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:00.475001 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.474974 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfld6\" (UniqueName: \"kubernetes.io/projected/33426b4e-740d-4fee-abf5-09793b47bdb9-kube-api-access-rfld6\") pod \"maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584\" (UID: \"33426b4e-740d-4fee-abf5-09793b47bdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:00.475398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.475369 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8684\" (UniqueName: \"kubernetes.io/projected/44553b66-1042-4dd2-95cb-76d526b887a1-kube-api-access-k8684\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf\" (UID: \"44553b66-1042-4dd2-95cb-76d526b887a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:00.475503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.475372 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s8nh\" (UniqueName: \"kubernetes.io/projected/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-kube-api-access-6s8nh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr\" (UID: \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:00.475630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.475611 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsnm\" (UniqueName: \"kubernetes.io/projected/85ef5cef-d3ab-4f83-861f-b64a421a90c0-kube-api-access-dhsnm\") pod \"maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd\" (UID: \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:00.475676 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.475643 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztp2k\" (UniqueName: \"kubernetes.io/projected/96eba2c2-e54d-4443-b017-e5f3e10b1742-kube-api-access-ztp2k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8\" (UID: \"96eba2c2-e54d-4443-b017-e5f3e10b1742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:00.550150 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.550102 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:00.554286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.554256 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:00.588702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.588669 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:00.601376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.601336 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:00.627343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.626882 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:00.634194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.633785 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:00.846145 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.846100 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8"] Nov 04 00:15:00.846312 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:15:00.846291 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3016b98a_6ae1_48e2_96bf_aebfd6f00f93.slice/crio-d327789d23e8234018b134e7d74fa4ffb3e5e85d837ceaa8928d47454670bdaf WatchSource:0}: Error finding container d327789d23e8234018b134e7d74fa4ffb3e5e85d837ceaa8928d47454670bdaf: Status 404 returned error can't find the container with id d327789d23e8234018b134e7d74fa4ffb3e5e85d837ceaa8928d47454670bdaf Nov 04 00:15:00.971207 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:15:00.971169 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33426b4e_740d_4fee_abf5_09793b47bdb9.slice/crio-7c5f141ae5adb58053746e38bfbe9afa912b2e395e23d926b5800018fa36822c WatchSource:0}: Error finding container 7c5f141ae5adb58053746e38bfbe9afa912b2e395e23d926b5800018fa36822c: Status 404 returned error can't find the container with id 7c5f141ae5adb58053746e38bfbe9afa912b2e395e23d926b5800018fa36822c Nov 04 00:15:00.971707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.971681 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584"] Nov 04 00:15:00.973934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.973910 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" event={"ID":"3016b98a-6ae1-48e2-96bf-aebfd6f00f93","Type":"ContainerStarted","Data":"d327789d23e8234018b134e7d74fa4ffb3e5e85d837ceaa8928d47454670bdaf"} Nov 04 00:15:00.986327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:00.986295 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr"] Nov 04 00:15:00.987348 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:15:00.987323 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99766a5_ce9e_43b4_ac97_da9f3b408d6f.slice/crio-6710063daf89f986f863dee5b2d0f78901dbbc69e4f62288fc7f49579e0375fb WatchSource:0}: Error finding container 6710063daf89f986f863dee5b2d0f78901dbbc69e4f62288fc7f49579e0375fb: Status 404 returned error can't find the container with id 6710063daf89f986f863dee5b2d0f78901dbbc69e4f62288fc7f49579e0375fb Nov 04 00:15:01.065431 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.065247 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8"] Nov 04 00:15:01.125685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.125657 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd"] Nov 04 00:15:01.131193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.131163 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf"] Nov 04 00:15:01.157584 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:15:01.157550 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96eba2c2_e54d_4443_b017_e5f3e10b1742.slice/crio-dff70ae7b5ab7a2ed72d9106a1353ecbe1f8b2a19e842365653731a9903cea1e WatchSource:0}: Error finding container dff70ae7b5ab7a2ed72d9106a1353ecbe1f8b2a19e842365653731a9903cea1e: Status 404 returned error can't find the container with id dff70ae7b5ab7a2ed72d9106a1353ecbe1f8b2a19e842365653731a9903cea1e Nov 04 00:15:01.158215 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:15:01.158182 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ef5cef_d3ab_4f83_861f_b64a421a90c0.slice/crio-a27c26fe5d39f8dec253a3ce70cf439f781dcc0c5e5d5eb4085216714d89bfe0 WatchSource:0}: Error finding container a27c26fe5d39f8dec253a3ce70cf439f781dcc0c5e5d5eb4085216714d89bfe0: Status 404 returned error can't find the container with id a27c26fe5d39f8dec253a3ce70cf439f781dcc0c5e5d5eb4085216714d89bfe0 Nov 04 00:15:01.159425 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:15:01.159401 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44553b66_1042_4dd2_95cb_76d526b887a1.slice/crio-285a864a047190d6b91d5fdd5e17cc8674ee311f443571340b174851741fec64 WatchSource:0}: Error finding container 285a864a047190d6b91d5fdd5e17cc8674ee311f443571340b174851741fec64: Status 404 returned error can't find the container with id 285a864a047190d6b91d5fdd5e17cc8674ee311f443571340b174851741fec64 Nov 04 00:15:01.267993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.267961 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:01.982774 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.982725 6208 generic.go:356] "Generic (PLEG): container finished" podID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerID="baf0c10fb65e0857e7f158e8034d0c1f6568bde9e233cfec95b0fe59150e2845" exitCode=60 Nov 04 00:15:01.982923 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.982781 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" event={"ID":"3016b98a-6ae1-48e2-96bf-aebfd6f00f93","Type":"ContainerDied","Data":"baf0c10fb65e0857e7f158e8034d0c1f6568bde9e233cfec95b0fe59150e2845"} Nov 04 00:15:01.983210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.983187 6208 scope.go:122] "RemoveContainer" containerID="baf0c10fb65e0857e7f158e8034d0c1f6568bde9e233cfec95b0fe59150e2845" Nov 04 00:15:01.984614 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.984590 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" event={"ID":"96eba2c2-e54d-4443-b017-e5f3e10b1742","Type":"ContainerStarted","Data":"e37467c386007ce3d0028b78b1439cf29b1f5aa44f20817990463bb7f5858309"} Nov 04 00:15:01.984726 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.984640 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" event={"ID":"96eba2c2-e54d-4443-b017-e5f3e10b1742","Type":"ContainerStarted","Data":"dff70ae7b5ab7a2ed72d9106a1353ecbe1f8b2a19e842365653731a9903cea1e"} Nov 04 00:15:01.986736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.986711 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" event={"ID":"44553b66-1042-4dd2-95cb-76d526b887a1","Type":"ContainerStarted","Data":"40c6c6ffc33517dfb66d2848602be945920ec30969354a44666728796a7c4591"} Nov 04 00:15:01.986834 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.986753 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" event={"ID":"44553b66-1042-4dd2-95cb-76d526b887a1","Type":"ContainerStarted","Data":"285a864a047190d6b91d5fdd5e17cc8674ee311f443571340b174851741fec64"} Nov 04 00:15:01.988492 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.988469 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" event={"ID":"33426b4e-740d-4fee-abf5-09793b47bdb9","Type":"ContainerStarted","Data":"04a0017a40bf4d1164176ee4150479ca744f2203832dc622262771f6c48c4306"} Nov 04 00:15:01.988609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.988516 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" event={"ID":"33426b4e-740d-4fee-abf5-09793b47bdb9","Type":"ContainerStarted","Data":"7c5f141ae5adb58053746e38bfbe9afa912b2e395e23d926b5800018fa36822c"} Nov 04 00:15:01.990084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.990060 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" event={"ID":"e99766a5-ce9e-43b4-ac97-da9f3b408d6f","Type":"ContainerStarted","Data":"e6531fc86e9bd99f5bef8b1aca4289fdeae05d4215000fead01b8c5068e7ccd1"} Nov 04 00:15:01.990201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.990108 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" event={"ID":"e99766a5-ce9e-43b4-ac97-da9f3b408d6f","Type":"ContainerStarted","Data":"6710063daf89f986f863dee5b2d0f78901dbbc69e4f62288fc7f49579e0375fb"} Nov 04 00:15:01.991698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.991674 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" event={"ID":"85ef5cef-d3ab-4f83-861f-b64a421a90c0","Type":"ContainerStarted","Data":"7874f14dbb885e52b688b6d1fc9ee4dcd429f5cd37c8f3f7651afa8ac4199adf"} Nov 04 00:15:01.991818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:01.991713 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" event={"ID":"85ef5cef-d3ab-4f83-861f-b64a421a90c0","Type":"ContainerStarted","Data":"a27c26fe5d39f8dec253a3ce70cf439f781dcc0c5e5d5eb4085216714d89bfe0"} Nov 04 00:15:02.021104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:02.021022 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" podStartSLOduration=2.021003431 podStartE2EDuration="2.021003431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:02.017430319 +0000 UTC m=+2720.840026483" watchObservedRunningTime="2026-11-04 00:15:02.021003431 +0000 UTC m=+2720.843599593" Nov 04 00:15:02.036183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:02.036130 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" podStartSLOduration=2.036115127 podStartE2EDuration="2.036115127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:02.035657783 +0000 UTC m=+2720.858253946" watchObservedRunningTime="2026-11-04 00:15:02.036115127 +0000 UTC m=+2720.858711289" Nov 04 00:15:02.061352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:02.061296 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" podStartSLOduration=2.061281672 podStartE2EDuration="2.061281672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:02.057228026 +0000 UTC m=+2720.879824190" watchObservedRunningTime="2026-11-04 00:15:02.061281672 +0000 UTC m=+2720.883877834" Nov 04 00:15:02.080342 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:02.080287 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" podStartSLOduration=2.080277031 podStartE2EDuration="2.080277031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:02.077175866 +0000 UTC m=+2720.899772027" watchObservedRunningTime="2026-11-04 00:15:02.080277031 +0000 UTC m=+2720.902873193" Nov 04 00:15:02.099364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:02.099312 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" podStartSLOduration=2.099297326 podStartE2EDuration="2.099297326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:02.095780135 +0000 UTC m=+2720.918376297" watchObservedRunningTime="2026-11-04 00:15:02.099297326 +0000 UTC m=+2720.921893492" Nov 04 00:15:02.131377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:02.131337 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:02.893241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:02.893153 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:03.019910 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.019874 6208 generic.go:356] "Generic (PLEG): container finished" podID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerID="7874f14dbb885e52b688b6d1fc9ee4dcd429f5cd37c8f3f7651afa8ac4199adf" exitCode=60 Nov 04 00:15:03.020144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.019950 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" event={"ID":"85ef5cef-d3ab-4f83-861f-b64a421a90c0","Type":"ContainerDied","Data":"7874f14dbb885e52b688b6d1fc9ee4dcd429f5cd37c8f3f7651afa8ac4199adf"} Nov 04 00:15:03.020358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.020340 6208 scope.go:122] "RemoveContainer" containerID="7874f14dbb885e52b688b6d1fc9ee4dcd429f5cd37c8f3f7651afa8ac4199adf" Nov 04 00:15:03.022388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.022359 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" event={"ID":"3016b98a-6ae1-48e2-96bf-aebfd6f00f93","Type":"ContainerStarted","Data":"3ab480fb5a1976eeccad6af05944d460c2a0753e5986741777acf857af5f1314"} Nov 04 00:15:03.024277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.024247 6208 generic.go:356] "Generic (PLEG): container finished" podID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerID="e37467c386007ce3d0028b78b1439cf29b1f5aa44f20817990463bb7f5858309" exitCode=60 Nov 04 00:15:03.024430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.024338 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" event={"ID":"96eba2c2-e54d-4443-b017-e5f3e10b1742","Type":"ContainerDied","Data":"e37467c386007ce3d0028b78b1439cf29b1f5aa44f20817990463bb7f5858309"} Nov 04 00:15:03.024533 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.024517 6208 scope.go:122] "RemoveContainer" containerID="e37467c386007ce3d0028b78b1439cf29b1f5aa44f20817990463bb7f5858309" Nov 04 00:15:03.025950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.025926 6208 generic.go:356] "Generic (PLEG): container finished" podID="44553b66-1042-4dd2-95cb-76d526b887a1" containerID="40c6c6ffc33517dfb66d2848602be945920ec30969354a44666728796a7c4591" exitCode=60 Nov 04 00:15:03.026088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.025999 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" event={"ID":"44553b66-1042-4dd2-95cb-76d526b887a1","Type":"ContainerDied","Data":"40c6c6ffc33517dfb66d2848602be945920ec30969354a44666728796a7c4591"} Nov 04 00:15:03.026252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.026236 6208 scope.go:122] "RemoveContainer" containerID="40c6c6ffc33517dfb66d2848602be945920ec30969354a44666728796a7c4591" Nov 04 00:15:03.027681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.027657 6208 generic.go:356] "Generic (PLEG): container finished" podID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerID="04a0017a40bf4d1164176ee4150479ca744f2203832dc622262771f6c48c4306" exitCode=60 Nov 04 00:15:03.027776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.027741 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" event={"ID":"33426b4e-740d-4fee-abf5-09793b47bdb9","Type":"ContainerDied","Data":"04a0017a40bf4d1164176ee4150479ca744f2203832dc622262771f6c48c4306"} Nov 04 00:15:03.027951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.027936 6208 scope.go:122] "RemoveContainer" containerID="04a0017a40bf4d1164176ee4150479ca744f2203832dc622262771f6c48c4306" Nov 04 00:15:03.029355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.029335 6208 generic.go:356] "Generic (PLEG): container finished" podID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerID="e6531fc86e9bd99f5bef8b1aca4289fdeae05d4215000fead01b8c5068e7ccd1" exitCode=60 Nov 04 00:15:03.029436 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.029377 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" event={"ID":"e99766a5-ce9e-43b4-ac97-da9f3b408d6f","Type":"ContainerDied","Data":"e6531fc86e9bd99f5bef8b1aca4289fdeae05d4215000fead01b8c5068e7ccd1"} Nov 04 00:15:03.029629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.029614 6208 scope.go:122] "RemoveContainer" containerID="e6531fc86e9bd99f5bef8b1aca4289fdeae05d4215000fead01b8c5068e7ccd1" Nov 04 00:15:03.060893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.060841 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" podStartSLOduration=3.060824923 podStartE2EDuration="3.060824923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:15:03.059493603 +0000 UTC m=+2721.882089765" watchObservedRunningTime="2026-11-04 00:15:03.060824923 +0000 UTC m=+2721.883421085" Nov 04 00:15:03.854711 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:03.854608 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:04.040115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.040017 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" event={"ID":"33426b4e-740d-4fee-abf5-09793b47bdb9","Type":"ContainerStarted","Data":"87550bbf4a48192c5e2769fe903b400256c9e8a51c523f913a728aec41c2bc46"} Nov 04 00:15:04.046717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.046678 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" event={"ID":"e99766a5-ce9e-43b4-ac97-da9f3b408d6f","Type":"ContainerStarted","Data":"dca6dba424aa9c7dba77e398460d956ffb0d5f91fa2cf2a65bc43c58d8c0801f"} Nov 04 00:15:04.049020 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.048984 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" event={"ID":"85ef5cef-d3ab-4f83-861f-b64a421a90c0","Type":"ContainerStarted","Data":"11096b89b279771eb5256195c736f86715310b8e4556cefb36a3d79f8ed03ae6"} Nov 04 00:15:04.051074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.051029 6208 generic.go:356] "Generic (PLEG): container finished" podID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerID="3ab480fb5a1976eeccad6af05944d460c2a0753e5986741777acf857af5f1314" exitCode=60 Nov 04 00:15:04.051189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.051140 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" event={"ID":"3016b98a-6ae1-48e2-96bf-aebfd6f00f93","Type":"ContainerDied","Data":"3ab480fb5a1976eeccad6af05944d460c2a0753e5986741777acf857af5f1314"} Nov 04 00:15:04.051268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.051199 6208 scope.go:122] "RemoveContainer" containerID="baf0c10fb65e0857e7f158e8034d0c1f6568bde9e233cfec95b0fe59150e2845" Nov 04 00:15:04.051394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.051371 6208 scope.go:122] "RemoveContainer" containerID="3ab480fb5a1976eeccad6af05944d460c2a0753e5986741777acf857af5f1314" Nov 04 00:15:04.051756 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:04.051735 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8_odh-ai-gateway-infra(3016b98a-6ae1-48e2-96bf-aebfd6f00f93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" Nov 04 00:15:04.059205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.059162 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" event={"ID":"96eba2c2-e54d-4443-b017-e5f3e10b1742","Type":"ContainerStarted","Data":"b46ebcccdfe017ad25795cadc6c318c53c3e5fab3f7586ff13e5d7a971256cec"} Nov 04 00:15:04.064661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.064623 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" event={"ID":"44553b66-1042-4dd2-95cb-76d526b887a1","Type":"ContainerStarted","Data":"50f607e72ece65d0c9a39c47e52c789c9d4513d9df84232769a7b549935cba83"} Nov 04 00:15:04.942054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:04.942005 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:05.078274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.078242 6208 generic.go:356] "Generic (PLEG): container finished" podID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerID="dca6dba424aa9c7dba77e398460d956ffb0d5f91fa2cf2a65bc43c58d8c0801f" exitCode=60 Nov 04 00:15:05.078739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.078390 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" event={"ID":"e99766a5-ce9e-43b4-ac97-da9f3b408d6f","Type":"ContainerDied","Data":"dca6dba424aa9c7dba77e398460d956ffb0d5f91fa2cf2a65bc43c58d8c0801f"} Nov 04 00:15:05.078739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.078454 6208 scope.go:122] "RemoveContainer" containerID="e6531fc86e9bd99f5bef8b1aca4289fdeae05d4215000fead01b8c5068e7ccd1" Nov 04 00:15:05.079755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.079734 6208 scope.go:122] "RemoveContainer" containerID="dca6dba424aa9c7dba77e398460d956ffb0d5f91fa2cf2a65bc43c58d8c0801f" Nov 04 00:15:05.080179 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:05.080156 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr_odh-ai-gateway-infra(e99766a5-ce9e-43b4-ac97-da9f3b408d6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" Nov 04 00:15:05.083919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.083490 6208 generic.go:356] "Generic (PLEG): container finished" podID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerID="11096b89b279771eb5256195c736f86715310b8e4556cefb36a3d79f8ed03ae6" exitCode=60 Nov 04 00:15:05.083919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.083628 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" event={"ID":"85ef5cef-d3ab-4f83-861f-b64a421a90c0","Type":"ContainerDied","Data":"11096b89b279771eb5256195c736f86715310b8e4556cefb36a3d79f8ed03ae6"} Nov 04 00:15:05.084581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.084560 6208 scope.go:122] "RemoveContainer" containerID="11096b89b279771eb5256195c736f86715310b8e4556cefb36a3d79f8ed03ae6" Nov 04 00:15:05.084942 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:05.084917 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd_odh-ai-gateway-infra(85ef5cef-d3ab-4f83-861f-b64a421a90c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" Nov 04 00:15:05.086656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.086638 6208 scope.go:122] "RemoveContainer" containerID="3ab480fb5a1976eeccad6af05944d460c2a0753e5986741777acf857af5f1314" Nov 04 00:15:05.086925 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:05.086902 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8_odh-ai-gateway-infra(3016b98a-6ae1-48e2-96bf-aebfd6f00f93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.088871 6208 generic.go:356] "Generic (PLEG): container finished" podID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerID="b46ebcccdfe017ad25795cadc6c318c53c3e5fab3f7586ff13e5d7a971256cec" exitCode=60 Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.088966 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" event={"ID":"96eba2c2-e54d-4443-b017-e5f3e10b1742","Type":"ContainerDied","Data":"b46ebcccdfe017ad25795cadc6c318c53c3e5fab3f7586ff13e5d7a971256cec"} Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.089552 6208 scope.go:122] "RemoveContainer" containerID="b46ebcccdfe017ad25795cadc6c318c53c3e5fab3f7586ff13e5d7a971256cec" Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:05.089885 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8_odh-ai-gateway-infra(96eba2c2-e54d-4443-b017-e5f3e10b1742)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.092422 6208 generic.go:356] "Generic (PLEG): container finished" podID="44553b66-1042-4dd2-95cb-76d526b887a1" containerID="50f607e72ece65d0c9a39c47e52c789c9d4513d9df84232769a7b549935cba83" exitCode=60 Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.092565 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" event={"ID":"44553b66-1042-4dd2-95cb-76d526b887a1","Type":"ContainerDied","Data":"50f607e72ece65d0c9a39c47e52c789c9d4513d9df84232769a7b549935cba83"} Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.093649 6208 scope.go:122] "RemoveContainer" containerID="50f607e72ece65d0c9a39c47e52c789c9d4513d9df84232769a7b549935cba83" Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:05.093972 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf_odh-ai-gateway-infra(44553b66-1042-4dd2-95cb-76d526b887a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.094983 6208 generic.go:356] "Generic (PLEG): container finished" podID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerID="87550bbf4a48192c5e2769fe903b400256c9e8a51c523f913a728aec41c2bc46" exitCode=60 Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.095053 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" event={"ID":"33426b4e-740d-4fee-abf5-09793b47bdb9","Type":"ContainerDied","Data":"87550bbf4a48192c5e2769fe903b400256c9e8a51c523f913a728aec41c2bc46"} Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.095381 6208 scope.go:122] "RemoveContainer" containerID="87550bbf4a48192c5e2769fe903b400256c9e8a51c523f913a728aec41c2bc46" Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:05.095691 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584_odh-ai-gateway-infra(33426b4e-740d-4fee-abf5-09793b47bdb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" Nov 04 00:15:05.103050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.100465 6208 scope.go:122] "RemoveContainer" containerID="7874f14dbb885e52b688b6d1fc9ee4dcd429f5cd37c8f3f7651afa8ac4199adf" Nov 04 00:15:05.118798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.118773 6208 scope.go:122] "RemoveContainer" containerID="e37467c386007ce3d0028b78b1439cf29b1f5aa44f20817990463bb7f5858309" Nov 04 00:15:05.133530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.133510 6208 scope.go:122] "RemoveContainer" containerID="40c6c6ffc33517dfb66d2848602be945920ec30969354a44666728796a7c4591" Nov 04 00:15:05.133616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.133570 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:05.147507 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.147484 6208 scope.go:122] "RemoveContainer" containerID="04a0017a40bf4d1164176ee4150479ca744f2203832dc622262771f6c48c4306" Nov 04 00:15:05.236338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.236303 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:05.337495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:05.337458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:05.769082 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:05.768220 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:15:06.105043 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:06.104943 6208 scope.go:122] "RemoveContainer" containerID="11096b89b279771eb5256195c736f86715310b8e4556cefb36a3d79f8ed03ae6" Nov 04 00:15:06.105481 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:06.105335 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd_odh-ai-gateway-infra(85ef5cef-d3ab-4f83-861f-b64a421a90c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" Nov 04 00:15:06.107153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:06.107130 6208 scope.go:122] "RemoveContainer" containerID="b46ebcccdfe017ad25795cadc6c318c53c3e5fab3f7586ff13e5d7a971256cec" Nov 04 00:15:06.107460 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:06.107429 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8_odh-ai-gateway-infra(96eba2c2-e54d-4443-b017-e5f3e10b1742)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" Nov 04 00:15:06.109410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:06.109390 6208 scope.go:122] "RemoveContainer" containerID="50f607e72ece65d0c9a39c47e52c789c9d4513d9df84232769a7b549935cba83" Nov 04 00:15:06.109698 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:06.109669 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf_odh-ai-gateway-infra(44553b66-1042-4dd2-95cb-76d526b887a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" Nov 04 00:15:06.111177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:06.111158 6208 scope.go:122] "RemoveContainer" containerID="87550bbf4a48192c5e2769fe903b400256c9e8a51c523f913a728aec41c2bc46" Nov 04 00:15:06.111398 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:06.111379 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584_odh-ai-gateway-infra(33426b4e-740d-4fee-abf5-09793b47bdb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" Nov 04 00:15:06.112690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:06.112673 6208 scope.go:122] "RemoveContainer" containerID="dca6dba424aa9c7dba77e398460d956ffb0d5f91fa2cf2a65bc43c58d8c0801f" Nov 04 00:15:06.112883 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:06.112867 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr_odh-ai-gateway-infra(e99766a5-ce9e-43b4-ac97-da9f3b408d6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" Nov 04 00:15:06.435006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:06.434921 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:07.295893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:07.295859 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:07.406868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:07.406835 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:07.819505 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:07.819365 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:08.138612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:08.138516 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:08.250067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:08.249995 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:09.059330 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:09.059292 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:09.838180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:09.838142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:10.844779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:10.844742 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:11.559297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:11.559256 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:11.659973 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:11.659923 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:11.759048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:11.758996 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:12.604955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:12.604918 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:12.880843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:12.880745 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:12.986264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:12.986234 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:13.636271 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:13.636231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:14.448611 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:14.448572 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:15.261102 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:15.261069 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:15.287937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:15.287911 6208 scope.go:122] "RemoveContainer" containerID="50f607e72ece65d0c9a39c47e52c789c9d4513d9df84232769a7b549935cba83" Nov 04 00:15:15.290089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:15.288544 6208 scope.go:122] "RemoveContainer" containerID="3ab480fb5a1976eeccad6af05944d460c2a0753e5986741777acf857af5f1314" Nov 04 00:15:15.290089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:15.289022 6208 scope.go:122] "RemoveContainer" containerID="dca6dba424aa9c7dba77e398460d956ffb0d5f91fa2cf2a65bc43c58d8c0801f" Nov 04 00:15:15.290089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:15.289458 6208 scope.go:122] "RemoveContainer" containerID="11096b89b279771eb5256195c736f86715310b8e4556cefb36a3d79f8ed03ae6" Nov 04 00:15:16.045640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:16.045603 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:16.300593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:16.300552 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" event={"ID":"85ef5cef-d3ab-4f83-861f-b64a421a90c0","Type":"ContainerStarted","Data":"617648fa1313cd463ac15915b3e96d815c54297fb38ed648afdc2b9c7b2476f8"} Nov 04 00:15:16.302725 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:16.302684 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" event={"ID":"3016b98a-6ae1-48e2-96bf-aebfd6f00f93","Type":"ContainerStarted","Data":"a9b0dbd0983b907c6e1e944eeb15dce974ab8a9c103227f0a3a7491ff3abeca1"} Nov 04 00:15:16.304557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:16.304531 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" event={"ID":"44553b66-1042-4dd2-95cb-76d526b887a1","Type":"ContainerStarted","Data":"2cb3ea01cbe40e3dc803195f308f6c5cf02f4f2df9d578ceaf04cdee098cc9ad"} Nov 04 00:15:16.306401 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:16.306380 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" event={"ID":"e99766a5-ce9e-43b4-ac97-da9f3b408d6f","Type":"ContainerStarted","Data":"145b868b91ebb4eec05bc4e8c1b68339116495087b4988a212cfeca35750adb2"} Nov 04 00:15:16.924302 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:16.924189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:17.315823 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.315789 6208 generic.go:356] "Generic (PLEG): container finished" podID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerID="617648fa1313cd463ac15915b3e96d815c54297fb38ed648afdc2b9c7b2476f8" exitCode=60 Nov 04 00:15:17.316283 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.315861 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" event={"ID":"85ef5cef-d3ab-4f83-861f-b64a421a90c0","Type":"ContainerDied","Data":"617648fa1313cd463ac15915b3e96d815c54297fb38ed648afdc2b9c7b2476f8"} Nov 04 00:15:17.316283 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.315923 6208 scope.go:122] "RemoveContainer" containerID="11096b89b279771eb5256195c736f86715310b8e4556cefb36a3d79f8ed03ae6" Nov 04 00:15:17.316378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.316330 6208 scope.go:122] "RemoveContainer" containerID="617648fa1313cd463ac15915b3e96d815c54297fb38ed648afdc2b9c7b2476f8" Nov 04 00:15:17.316704 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:17.316677 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd_odh-ai-gateway-infra(85ef5cef-d3ab-4f83-861f-b64a421a90c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" Nov 04 00:15:17.318279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.318258 6208 generic.go:356] "Generic (PLEG): container finished" podID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerID="a9b0dbd0983b907c6e1e944eeb15dce974ab8a9c103227f0a3a7491ff3abeca1" exitCode=60 Nov 04 00:15:17.318385 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.318310 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" event={"ID":"3016b98a-6ae1-48e2-96bf-aebfd6f00f93","Type":"ContainerDied","Data":"a9b0dbd0983b907c6e1e944eeb15dce974ab8a9c103227f0a3a7491ff3abeca1"} Nov 04 00:15:17.318674 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.318659 6208 scope.go:122] "RemoveContainer" containerID="a9b0dbd0983b907c6e1e944eeb15dce974ab8a9c103227f0a3a7491ff3abeca1" Nov 04 00:15:17.318973 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:17.318955 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8_odh-ai-gateway-infra(3016b98a-6ae1-48e2-96bf-aebfd6f00f93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" Nov 04 00:15:17.320282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.320260 6208 generic.go:356] "Generic (PLEG): container finished" podID="44553b66-1042-4dd2-95cb-76d526b887a1" containerID="2cb3ea01cbe40e3dc803195f308f6c5cf02f4f2df9d578ceaf04cdee098cc9ad" exitCode=60 Nov 04 00:15:17.320383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.320328 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" event={"ID":"44553b66-1042-4dd2-95cb-76d526b887a1","Type":"ContainerDied","Data":"2cb3ea01cbe40e3dc803195f308f6c5cf02f4f2df9d578ceaf04cdee098cc9ad"} Nov 04 00:15:17.320777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.320746 6208 scope.go:122] "RemoveContainer" containerID="2cb3ea01cbe40e3dc803195f308f6c5cf02f4f2df9d578ceaf04cdee098cc9ad" Nov 04 00:15:17.321122 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:17.321101 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf_odh-ai-gateway-infra(44553b66-1042-4dd2-95cb-76d526b887a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" Nov 04 00:15:17.322868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.322846 6208 generic.go:356] "Generic (PLEG): container finished" podID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerID="145b868b91ebb4eec05bc4e8c1b68339116495087b4988a212cfeca35750adb2" exitCode=60 Nov 04 00:15:17.322992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.322904 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" event={"ID":"e99766a5-ce9e-43b4-ac97-da9f3b408d6f","Type":"ContainerDied","Data":"145b868b91ebb4eec05bc4e8c1b68339116495087b4988a212cfeca35750adb2"} Nov 04 00:15:17.323218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.323202 6208 scope.go:122] "RemoveContainer" containerID="145b868b91ebb4eec05bc4e8c1b68339116495087b4988a212cfeca35750adb2" Nov 04 00:15:17.323506 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:17.323489 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr_odh-ai-gateway-infra(e99766a5-ce9e-43b4-ac97-da9f3b408d6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" Nov 04 00:15:17.332157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.332137 6208 scope.go:122] "RemoveContainer" containerID="3ab480fb5a1976eeccad6af05944d460c2a0753e5986741777acf857af5f1314" Nov 04 00:15:17.333939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.333920 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd"] Nov 04 00:15:17.347711 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.347522 6208 scope.go:122] "RemoveContainer" containerID="50f607e72ece65d0c9a39c47e52c789c9d4513d9df84232769a7b549935cba83" Nov 04 00:15:17.351799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.351778 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr"] Nov 04 00:15:17.362890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.362870 6208 scope.go:122] "RemoveContainer" containerID="dca6dba424aa9c7dba77e398460d956ffb0d5f91fa2cf2a65bc43c58d8c0801f" Nov 04 00:15:17.375885 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.375856 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf"] Nov 04 00:15:17.392232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.392205 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8"] Nov 04 00:15:17.738740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:17.738704 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:18.287630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.287603 6208 scope.go:122] "RemoveContainer" containerID="87550bbf4a48192c5e2769fe903b400256c9e8a51c523f913a728aec41c2bc46" Nov 04 00:15:18.287814 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.287757 6208 scope.go:122] "RemoveContainer" containerID="b46ebcccdfe017ad25795cadc6c318c53c3e5fab3f7586ff13e5d7a971256cec" Nov 04 00:15:18.531813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.531775 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:18.725416 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.725385 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:18.779666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.779629 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:18.781836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.781810 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:18.786474 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.786451 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:18.873946 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.873921 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-kube-api-access-6s8nh\" (UniqueName: \"kubernetes.io/projected/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-kube-api-access-6s8nh\") pod \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\" (UID: \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\") " Nov 04 00:15:18.874065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874002 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-kube-api-access-8fzb4\" (UniqueName: \"kubernetes.io/projected/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-kube-api-access-8fzb4\") pod \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\" (UID: \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\") " Nov 04 00:15:18.874135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874102 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-maas-api-service-ca\") pod \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\" (UID: \"e99766a5-ce9e-43b4-ac97-da9f3b408d6f\") " Nov 04 00:15:18.874221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874207 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-maas-api-service-ca\") pod \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\" (UID: \"3016b98a-6ae1-48e2-96bf-aebfd6f00f93\") " Nov 04 00:15:18.874296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874286 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44553b66-1042-4dd2-95cb-76d526b887a1-kube-api-access-k8684\" (UniqueName: \"kubernetes.io/projected/44553b66-1042-4dd2-95cb-76d526b887a1-kube-api-access-k8684\") pod \"44553b66-1042-4dd2-95cb-76d526b887a1\" (UID: \"44553b66-1042-4dd2-95cb-76d526b887a1\") " Nov 04 00:15:18.874444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874423 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44553b66-1042-4dd2-95cb-76d526b887a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44553b66-1042-4dd2-95cb-76d526b887a1-maas-api-service-ca\") pod \"44553b66-1042-4dd2-95cb-76d526b887a1\" (UID: \"44553b66-1042-4dd2-95cb-76d526b887a1\") " Nov 04 00:15:18.874523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874474 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ef5cef-d3ab-4f83-861f-b64a421a90c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ef5cef-d3ab-4f83-861f-b64a421a90c0-maas-api-service-ca\") pod \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\" (UID: \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\") " Nov 04 00:15:18.874523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874510 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-maas-api-service-ca" pod "e99766a5-ce9e-43b4-ac97-da9f3b408d6f" (UID: "e99766a5-ce9e-43b4-ac97-da9f3b408d6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:15:18.874776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874534 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ef5cef-d3ab-4f83-861f-b64a421a90c0-kube-api-access-dhsnm\" (UniqueName: \"kubernetes.io/projected/85ef5cef-d3ab-4f83-861f-b64a421a90c0-kube-api-access-dhsnm\") pod \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\" (UID: \"85ef5cef-d3ab-4f83-861f-b64a421a90c0\") " Nov 04 00:15:18.874854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874818 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-maas-api-service-ca" pod "3016b98a-6ae1-48e2-96bf-aebfd6f00f93" (UID: "3016b98a-6ae1-48e2-96bf-aebfd6f00f93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:15:18.874916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.874823 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ef5cef-d3ab-4f83-861f-b64a421a90c0-maas-api-service-ca" pod "85ef5cef-d3ab-4f83-861f-b64a421a90c0" (UID: "85ef5cef-d3ab-4f83-861f-b64a421a90c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:15:18.875417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.875323 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ef5cef-d3ab-4f83-861f-b64a421a90c0-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:18.875417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.875354 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:18.875417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.875371 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:18.875842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.875807 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44553b66-1042-4dd2-95cb-76d526b887a1-maas-api-service-ca" pod "44553b66-1042-4dd2-95cb-76d526b887a1" (UID: "44553b66-1042-4dd2-95cb-76d526b887a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:15:18.876612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.876585 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-kube-api-access-6s8nh" pod "e99766a5-ce9e-43b4-ac97-da9f3b408d6f" (UID: "e99766a5-ce9e-43b4-ac97-da9f3b408d6f"). InnerVolumeSpecName "kube-api-access-6s8nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:15:18.876913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.876755 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-kube-api-access-8fzb4" pod "3016b98a-6ae1-48e2-96bf-aebfd6f00f93" (UID: "3016b98a-6ae1-48e2-96bf-aebfd6f00f93"). InnerVolumeSpecName "kube-api-access-8fzb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:15:18.876913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.876858 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ef5cef-d3ab-4f83-861f-b64a421a90c0-kube-api-access-dhsnm" pod "85ef5cef-d3ab-4f83-861f-b64a421a90c0" (UID: "85ef5cef-d3ab-4f83-861f-b64a421a90c0"). InnerVolumeSpecName "kube-api-access-dhsnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:15:18.877522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.877494 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44553b66-1042-4dd2-95cb-76d526b887a1-kube-api-access-k8684" pod "44553b66-1042-4dd2-95cb-76d526b887a1" (UID: "44553b66-1042-4dd2-95cb-76d526b887a1"). InnerVolumeSpecName "kube-api-access-k8684". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:15:18.976559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.976522 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8684\" (UniqueName: \"kubernetes.io/projected/44553b66-1042-4dd2-95cb-76d526b887a1-kube-api-access-k8684\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:18.976559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.976553 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44553b66-1042-4dd2-95cb-76d526b887a1-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:18.976559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.976564 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhsnm\" (UniqueName: \"kubernetes.io/projected/85ef5cef-d3ab-4f83-861f-b64a421a90c0-kube-api-access-dhsnm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:18.976801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.976573 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s8nh\" (UniqueName: \"kubernetes.io/projected/e99766a5-ce9e-43b4-ac97-da9f3b408d6f-kube-api-access-6s8nh\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:18.976801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:18.976582 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fzb4\" (UniqueName: \"kubernetes.io/projected/3016b98a-6ae1-48e2-96bf-aebfd6f00f93-kube-api-access-8fzb4\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:19.357377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.357329 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" event={"ID":"33426b4e-740d-4fee-abf5-09793b47bdb9","Type":"ContainerStarted","Data":"87dc18ee28383796716667ee38de98509d8a02586841a4ca6551399b92bf9734"} Nov 04 00:15:19.359201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.359169 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" Nov 04 00:15:19.359337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.359223 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr" event={"ID":"e99766a5-ce9e-43b4-ac97-da9f3b408d6f","Type":"ContainerDied","Data":"6710063daf89f986f863dee5b2d0f78901dbbc69e4f62288fc7f49579e0375fb"} Nov 04 00:15:19.359337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.359290 6208 scope.go:122] "RemoveContainer" containerID="145b868b91ebb4eec05bc4e8c1b68339116495087b4988a212cfeca35750adb2" Nov 04 00:15:19.361382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.360924 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" event={"ID":"85ef5cef-d3ab-4f83-861f-b64a421a90c0","Type":"ContainerDied","Data":"a27c26fe5d39f8dec253a3ce70cf439f781dcc0c5e5d5eb4085216714d89bfe0"} Nov 04 00:15:19.361382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.361017 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd" Nov 04 00:15:19.363919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.363895 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" event={"ID":"3016b98a-6ae1-48e2-96bf-aebfd6f00f93","Type":"ContainerDied","Data":"d327789d23e8234018b134e7d74fa4ffb3e5e85d837ceaa8928d47454670bdaf"} Nov 04 00:15:19.364058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.363979 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8" Nov 04 00:15:19.367006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.366983 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" event={"ID":"96eba2c2-e54d-4443-b017-e5f3e10b1742","Type":"ContainerStarted","Data":"35eb7760b8fe5e40ecca48dcd98e030903a28985fcfdf0bf79e67f6fb615348f"} Nov 04 00:15:19.368700 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.368678 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" event={"ID":"44553b66-1042-4dd2-95cb-76d526b887a1","Type":"ContainerDied","Data":"285a864a047190d6b91d5fdd5e17cc8674ee311f443571340b174851741fec64"} Nov 04 00:15:19.368786 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.368683 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf" Nov 04 00:15:19.374544 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.374526 6208 scope.go:122] "RemoveContainer" containerID="617648fa1313cd463ac15915b3e96d815c54297fb38ed648afdc2b9c7b2476f8" Nov 04 00:15:19.386420 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.386398 6208 scope.go:122] "RemoveContainer" containerID="a9b0dbd0983b907c6e1e944eeb15dce974ab8a9c103227f0a3a7491ff3abeca1" Nov 04 00:15:19.396277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.396256 6208 scope.go:122] "RemoveContainer" containerID="2cb3ea01cbe40e3dc803195f308f6c5cf02f4f2df9d578ceaf04cdee098cc9ad" Nov 04 00:15:19.414676 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.414653 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd"] Nov 04 00:15:19.426703 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.426678 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5cb349-29895855-fqhmd"] Nov 04 00:15:19.441300 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.441276 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8"] Nov 04 00:15:19.442249 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.442220 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:19.448296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.448277 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-2b1f66-29895855-jf8c8"] Nov 04 00:15:19.463919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.463897 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr"] Nov 04 00:15:19.467784 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.467760 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895855-wmklr"] Nov 04 00:15:19.557139 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.555293 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf"] Nov 04 00:15:19.560932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:19.560903 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-15b02a-29895855-rzvxf"] Nov 04 00:15:20.380552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.380518 6208 generic.go:356] "Generic (PLEG): container finished" podID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerID="35eb7760b8fe5e40ecca48dcd98e030903a28985fcfdf0bf79e67f6fb615348f" exitCode=60 Nov 04 00:15:20.380765 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.380591 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" event={"ID":"96eba2c2-e54d-4443-b017-e5f3e10b1742","Type":"ContainerDied","Data":"35eb7760b8fe5e40ecca48dcd98e030903a28985fcfdf0bf79e67f6fb615348f"} Nov 04 00:15:20.380765 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.380651 6208 scope.go:122] "RemoveContainer" containerID="b46ebcccdfe017ad25795cadc6c318c53c3e5fab3f7586ff13e5d7a971256cec" Nov 04 00:15:20.381175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.381154 6208 scope.go:122] "RemoveContainer" containerID="35eb7760b8fe5e40ecca48dcd98e030903a28985fcfdf0bf79e67f6fb615348f" Nov 04 00:15:20.381570 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:20.381547 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8_odh-ai-gateway-infra(96eba2c2-e54d-4443-b017-e5f3e10b1742)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" Nov 04 00:15:20.383943 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.383912 6208 generic.go:356] "Generic (PLEG): container finished" podID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerID="87dc18ee28383796716667ee38de98509d8a02586841a4ca6551399b92bf9734" exitCode=60 Nov 04 00:15:20.384087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.384064 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" event={"ID":"33426b4e-740d-4fee-abf5-09793b47bdb9","Type":"ContainerDied","Data":"87dc18ee28383796716667ee38de98509d8a02586841a4ca6551399b92bf9734"} Nov 04 00:15:20.384650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.384633 6208 scope.go:122] "RemoveContainer" containerID="87dc18ee28383796716667ee38de98509d8a02586841a4ca6551399b92bf9734" Nov 04 00:15:20.384981 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:20.384959 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584_odh-ai-gateway-infra(33426b4e-740d-4fee-abf5-09793b47bdb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" Nov 04 00:15:20.388876 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.388856 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584"] Nov 04 00:15:20.394552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.394534 6208 scope.go:122] "RemoveContainer" containerID="87550bbf4a48192c5e2769fe903b400256c9e8a51c523f913a728aec41c2bc46" Nov 04 00:15:20.412477 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.412436 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8"] Nov 04 00:15:20.496311 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:20.494400 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:21.299840 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.299806 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" path="/var/lib/kubelet/pods/3016b98a-6ae1-48e2-96bf-aebfd6f00f93/volumes" Nov 04 00:15:21.300384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.300367 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" path="/var/lib/kubelet/pods/44553b66-1042-4dd2-95cb-76d526b887a1/volumes" Nov 04 00:15:21.300777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.300763 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" path="/var/lib/kubelet/pods/85ef5cef-d3ab-4f83-861f-b64a421a90c0/volumes" Nov 04 00:15:21.301515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.301499 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" path="/var/lib/kubelet/pods/e99766a5-ce9e-43b4-ac97-da9f3b408d6f/volumes" Nov 04 00:15:21.465029 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.464995 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:21.651511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.651416 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:21.673687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.673653 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33426b4e-740d-4fee-abf5-09793b47bdb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33426b4e-740d-4fee-abf5-09793b47bdb9-maas-api-service-ca\") pod \"33426b4e-740d-4fee-abf5-09793b47bdb9\" (UID: \"33426b4e-740d-4fee-abf5-09793b47bdb9\") " Nov 04 00:15:21.673876 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.673824 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33426b4e-740d-4fee-abf5-09793b47bdb9-kube-api-access-rfld6\" (UniqueName: \"kubernetes.io/projected/33426b4e-740d-4fee-abf5-09793b47bdb9-kube-api-access-rfld6\") pod \"33426b4e-740d-4fee-abf5-09793b47bdb9\" (UID: \"33426b4e-740d-4fee-abf5-09793b47bdb9\") " Nov 04 00:15:21.675352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.675328 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33426b4e-740d-4fee-abf5-09793b47bdb9-maas-api-service-ca" pod "33426b4e-740d-4fee-abf5-09793b47bdb9" (UID: "33426b4e-740d-4fee-abf5-09793b47bdb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:15:21.687595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.687562 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33426b4e-740d-4fee-abf5-09793b47bdb9-kube-api-access-rfld6" pod "33426b4e-740d-4fee-abf5-09793b47bdb9" (UID: "33426b4e-740d-4fee-abf5-09793b47bdb9"). InnerVolumeSpecName "kube-api-access-rfld6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:15:21.719900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.719873 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:21.775951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.775909 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96eba2c2-e54d-4443-b017-e5f3e10b1742-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eba2c2-e54d-4443-b017-e5f3e10b1742-maas-api-service-ca\") pod \"96eba2c2-e54d-4443-b017-e5f3e10b1742\" (UID: \"96eba2c2-e54d-4443-b017-e5f3e10b1742\") " Nov 04 00:15:21.776761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.776015 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96eba2c2-e54d-4443-b017-e5f3e10b1742-kube-api-access-ztp2k\" (UniqueName: \"kubernetes.io/projected/96eba2c2-e54d-4443-b017-e5f3e10b1742-kube-api-access-ztp2k\") pod \"96eba2c2-e54d-4443-b017-e5f3e10b1742\" (UID: \"96eba2c2-e54d-4443-b017-e5f3e10b1742\") " Nov 04 00:15:21.776761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.776404 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96eba2c2-e54d-4443-b017-e5f3e10b1742-maas-api-service-ca" pod "96eba2c2-e54d-4443-b017-e5f3e10b1742" (UID: "96eba2c2-e54d-4443-b017-e5f3e10b1742"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:15:21.776923 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.776832 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33426b4e-740d-4fee-abf5-09793b47bdb9-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:21.776923 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.776850 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfld6\" (UniqueName: \"kubernetes.io/projected/33426b4e-740d-4fee-abf5-09793b47bdb9-kube-api-access-rfld6\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:21.776923 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.776861 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96eba2c2-e54d-4443-b017-e5f3e10b1742-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:21.778713 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.778687 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96eba2c2-e54d-4443-b017-e5f3e10b1742-kube-api-access-ztp2k" pod "96eba2c2-e54d-4443-b017-e5f3e10b1742" (UID: "96eba2c2-e54d-4443-b017-e5f3e10b1742"). InnerVolumeSpecName "kube-api-access-ztp2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:15:21.878390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:21.878293 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztp2k\" (UniqueName: \"kubernetes.io/projected/96eba2c2-e54d-4443-b017-e5f3e10b1742-kube-api-access-ztp2k\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:15:22.407641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.407612 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" Nov 04 00:15:22.408158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.407638 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8" event={"ID":"96eba2c2-e54d-4443-b017-e5f3e10b1742","Type":"ContainerDied","Data":"dff70ae7b5ab7a2ed72d9106a1353ecbe1f8b2a19e842365653731a9903cea1e"} Nov 04 00:15:22.408158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.407711 6208 scope.go:122] "RemoveContainer" containerID="35eb7760b8fe5e40ecca48dcd98e030903a28985fcfdf0bf79e67f6fb615348f" Nov 04 00:15:22.409588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.409561 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" event={"ID":"33426b4e-740d-4fee-abf5-09793b47bdb9","Type":"ContainerDied","Data":"7c5f141ae5adb58053746e38bfbe9afa912b2e395e23d926b5800018fa36822c"} Nov 04 00:15:22.409588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.409581 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584" Nov 04 00:15:22.421732 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.421708 6208 scope.go:122] "RemoveContainer" containerID="87dc18ee28383796716667ee38de98509d8a02586841a4ca6551399b92bf9734" Nov 04 00:15:22.437290 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.437261 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8"] Nov 04 00:15:22.439954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.439930 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:22.441834 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.441813 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4331a7-29895855-qtcz8"] Nov 04 00:15:22.462071 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.462046 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584"] Nov 04 00:15:22.471501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:22.471478 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c60893-29895855-7g584"] Nov 04 00:15:23.254667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:23.254627 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:23.376403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:23.376361 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" path="/var/lib/kubelet/pods/33426b4e-740d-4fee-abf5-09793b47bdb9/volumes" Nov 04 00:15:23.377203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:23.377172 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" path="/var/lib/kubelet/pods/96eba2c2-e54d-4443-b017-e5f3e10b1742/volumes" Nov 04 00:15:24.241666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:24.241627 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:25.269383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:25.268967 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:26.338087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:26.337625 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:26.719112 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:26.719077 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:27.746963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:27.746905 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:28.557297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:28.557260 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:29.019261 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:29.019224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:29.119289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:29.119249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:30.341990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:30.341954 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:31.154323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:31.154285 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:32.231958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:32.231871 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:32.334933 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:32.334895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:33.228361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:33.228330 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:33.958624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:33.958566 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:34.723274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:34.723228 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:35.524022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:35.523977 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:35.651978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:35.651935 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:35.708356 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:15:35.708320 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:15:35.750731 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:35.750692 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:36.463510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:36.463457 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:37.221760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:37.221708 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:37.626895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:37.626805 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:38.139420 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:38.139387 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:38.933249 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:38.933218 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:39.730886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:39.730852 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:39.853885 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:39.853847 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:39.950836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:39.950795 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:40.718301 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:40.718258 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:41.586077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:41.585712 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:42.330483 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:42.330446 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:43.042962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:43.042929 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:43.829514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:43.829470 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:44.637890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:44.637855 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:45.376347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:45.376304 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:46.229634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:46.229598 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:47.037647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:47.037604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:47.775930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:47.775890 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:48.275960 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:48.275916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:48.638978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:48.638876 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:49.433199 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:49.433140 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:50.323844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:50.323808 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:51.238722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:51.238676 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:51.555213 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:51.555128 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:51.656437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:51.656402 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:52.195263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:52.195233 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:52.412828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:52.412769 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:53.124872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:53.124835 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:54.053873 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:54.053836 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:54.939879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:54.939837 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:55.039309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:55.039277 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:55.147978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:55.147942 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:55.489014 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:55.488978 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:55.972623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:55.972585 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:56.756791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:56.756755 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:57.547444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:57.547393 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:58.333420 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:58.333388 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:59.111093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:59.111060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:59.222091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:59.222064 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:59.328784 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:59.328746 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:59.695412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:59.695377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:15:59.907800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:15:59.907761 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:00.113322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:00.113280 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:00.852075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:00.852008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:01.723930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:01.723893 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:02.532539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:02.532490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:02.633825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:02.633785 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:03.116056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.112182 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:03.487585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.486592 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.595694 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj"] Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596749 6208 cpu_manager.go:414] "containerMap: removing container" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596774 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596794 6208 cpu_manager.go:414] "containerMap: removing container" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596801 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596823 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596832 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596849 6208 cpu_manager.go:414] "containerMap: removing container" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596857 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596868 6208 cpu_manager.go:414] "containerMap: removing container" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596874 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596908 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596915 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596929 6208 cpu_manager.go:414] "containerMap: removing container" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596938 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596962 6208 cpu_manager.go:414] "containerMap: removing container" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596969 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596989 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.596997 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597011 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597018 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597052 6208 cpu_manager.go:414] "containerMap: removing container" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597060 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597083 6208 cpu_manager.go:414] "containerMap: removing container" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597090 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597115 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597121 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597284 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597295 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597305 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597314 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597321 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597334 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597345 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597402 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597414 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597423 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597436 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597466 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597478 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99766a5-ce9e-43b4-ac97-da9f3b408d6f" containerName="cleanup" Nov 04 00:16:03.600056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.597490 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerName="cleanup" Nov 04 00:16:03.658482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.658242 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:03.667875 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.666762 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj"] Nov 04 00:16:03.667875 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.666891 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.696750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.696342 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8cc1c0-openshift-default-dockercfg-nxf65\"" Nov 04 00:16:03.721177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.721140 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.721362 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.721200 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.721362 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.721302 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.721480 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.721441 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.721541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.721521 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.721687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.721670 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156db56-b84d-4241-9689-69a6aa1eaf5e-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.721745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.721726 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.721838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.721822 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.721962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.721943 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mm9\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-kube-api-access-t2mm9\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.776000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.775874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:03.824392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.824332 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mm9\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-kube-api-access-t2mm9\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.824629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.824471 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.824629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.824513 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.824629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.824569 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.824853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.824648 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.824853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.824692 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.824853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.824815 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156db56-b84d-4241-9689-69a6aa1eaf5e-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.825169 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.824868 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.825169 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.824926 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.825739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.825702 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.826475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.826446 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.842115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.839390 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.842115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.840383 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156db56-b84d-4241-9689-69a6aa1eaf5e-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.842115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.840632 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.870767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.845682 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.883429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.880866 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.897335 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.897259 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mm9\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-kube-api-access-t2mm9\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:03.934651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:03.934583 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:04.016313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:04.016015 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:04.387572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:04.387536 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj"] Nov 04 00:16:04.388068 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:16:04.388019 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1156db56_b84d_4241_9689_69a6aa1eaf5e.slice/crio-7d1f8097eea981a6fbe4306e827ff1c4a8e4635835fe4beecc561784b2801cfb WatchSource:0}: Error finding container 7d1f8097eea981a6fbe4306e827ff1c4a8e4635835fe4beecc561784b2801cfb: Status 404 returned error can't find the container with id 7d1f8097eea981a6fbe4306e827ff1c4a8e4635835fe4beecc561784b2801cfb Nov 04 00:16:04.391114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:04.391079 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:16:04.391208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:04.391150 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:16:04.391208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:04.391178 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:16:04.530541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:04.530502 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:05.035818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:05.035782 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" event={"ID":"1156db56-b84d-4241-9689-69a6aa1eaf5e","Type":"ContainerStarted","Data":"48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689"} Nov 04 00:16:05.036226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:05.035859 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" event={"ID":"1156db56-b84d-4241-9689-69a6aa1eaf5e","Type":"ContainerStarted","Data":"7d1f8097eea981a6fbe4306e827ff1c4a8e4635835fe4beecc561784b2801cfb"} Nov 04 00:16:05.067588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:05.067525 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podStartSLOduration=2.067508602 podStartE2EDuration="2.067508602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:05.064375693 +0000 UTC m=+2783.886971882" watchObservedRunningTime="2026-11-04 00:16:05.067508602 +0000 UTC m=+2783.890104765" Nov 04 00:16:05.334204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:05.334101 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:05.708517 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:16:05.708479 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:16:06.017643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:06.017541 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:06.024762 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:06.024734 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:06.043212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:06.043186 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:06.044164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:06.044146 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:16:06.055128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:06.055086 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:06.836461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:06.836427 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:06.936000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:06.935963 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:07.402048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:07.401995 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:07.503212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:07.503165 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:08.760715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:08.760676 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:08.938543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:08.938505 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:09.036684 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:09.036592 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:09.735899 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:09.735863 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:09.848788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:09.847420 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:10.492879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:10.492833 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:10.593405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:10.593371 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:10.701057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:10.700607 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:11.570470 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:11.570432 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:12.090668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:12.090629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:12.272115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:12.272077 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:12.995175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:12.995133 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:13.094555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:13.094524 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:13.440337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:13.440246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:13.706132 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:13.705378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:13.955970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:13.952579 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:14.055903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:14.055818 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:14.806644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:14.806609 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:15.554539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:15.554501 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:15.713279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:15.713240 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:16.360870 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:16.360835 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:17.164067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:17.163990 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:17.959868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:17.959831 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:18.747887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:18.747852 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:19.541884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:19.541849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:20.356016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:20.355981 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:20.553894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:20.553859 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:20.656976 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:20.656878 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:21.058371 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:21.058318 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:21.524541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:21.524498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:21.784513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:21.784419 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:22.084118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:22.082413 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:22.429241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:22.429150 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:23.213409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:23.213372 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:23.790364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:23.790325 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:23.894882 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:23.893678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:24.008865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.008825 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:24.339822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.339784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:24.755791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.755754 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92"] Nov 04 00:16:24.756783 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756763 6208 cpu_manager.go:414] "containerMap: removing container" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerName="cleanup" Nov 04 00:16:24.756867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756785 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerName="cleanup" Nov 04 00:16:24.756867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756818 6208 cpu_manager.go:414] "containerMap: removing container" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" containerName="cleanup" Nov 04 00:16:24.756867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756823 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" containerName="cleanup" Nov 04 00:16:24.756867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756838 6208 cpu_manager.go:414] "containerMap: removing container" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerName="cleanup" Nov 04 00:16:24.756867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756843 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerName="cleanup" Nov 04 00:16:24.757126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756872 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerName="cleanup" Nov 04 00:16:24.757126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756878 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3016b98a-6ae1-48e2-96bf-aebfd6f00f93" containerName="cleanup" Nov 04 00:16:24.757126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756888 6208 cpu_manager.go:414] "containerMap: removing container" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerName="cleanup" Nov 04 00:16:24.757126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756892 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerName="cleanup" Nov 04 00:16:24.757126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756989 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="33426b4e-740d-4fee-abf5-09793b47bdb9" containerName="cleanup" Nov 04 00:16:24.757126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.756997 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="44553b66-1042-4dd2-95cb-76d526b887a1" containerName="cleanup" Nov 04 00:16:24.757126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.757006 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="96eba2c2-e54d-4443-b017-e5f3e10b1742" containerName="cleanup" Nov 04 00:16:24.757126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.757016 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ef5cef-d3ab-4f83-861f-b64a421a90c0" containerName="cleanup" Nov 04 00:16:24.761860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.761838 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.772857 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.772827 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-954cd9-openshift-default-dockercfg-qcz6k\"" Nov 04 00:16:24.779935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.779904 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92"] Nov 04 00:16:24.790569 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.790517 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:24.879595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.879551 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-certs\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.879799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.879700 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-envoy\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.879799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.879790 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555966c0-f453-4441-8991-cf25c1200236-istiod-ca-cert\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.879936 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.879843 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-data\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.879936 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.879871 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-istio-token\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.880013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.879989 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mg2\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-kube-api-access-d2mg2\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.880146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.880126 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-credential-socket\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.880245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.880228 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-socket\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.880358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.880343 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555966c0-f453-4441-8991-cf25c1200236-istio-podinfo\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.897376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.897341 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:24.982592 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.982554 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-data\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.983233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.983206 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-data\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.983438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.983238 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-istio-token\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.983609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.983589 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mg2\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-kube-api-access-d2mg2\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.983816 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.983797 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-credential-socket\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.984066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.984044 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-socket\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.984456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.984427 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555966c0-f453-4441-8991-cf25c1200236-istio-podinfo\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.984566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.984444 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-socket\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.984672 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.984624 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-credential-socket\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.984881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.984794 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-certs\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.985978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.985353 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-envoy\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.985978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.985471 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555966c0-f453-4441-8991-cf25c1200236-istiod-ca-cert\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.985978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.985454 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-certs\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.986953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.986916 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555966c0-f453-4441-8991-cf25c1200236-istiod-ca-cert\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.992623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.992595 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-envoy\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:24.997398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.997311 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:24.998552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:24.998507 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mg2\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-kube-api-access-d2mg2\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:25.038050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:25.037929 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555966c0-f453-4441-8991-cf25c1200236-istio-podinfo\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:25.043496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:25.039211 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-istio-token\") pod \"e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:25.124172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:25.104322 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:25.572014 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:25.571983 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92"] Nov 04 00:16:25.572586 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:16:25.572561 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555966c0_f453_4441_8991_cf25c1200236.slice/crio-d16a192095da4c8b5cc0bb61d59ccfbaf03a506b130a82a15ee0e06211e911dd WatchSource:0}: Error finding container d16a192095da4c8b5cc0bb61d59ccfbaf03a506b130a82a15ee0e06211e911dd: Status 404 returned error can't find the container with id d16a192095da4c8b5cc0bb61d59ccfbaf03a506b130a82a15ee0e06211e911dd Nov 04 00:16:25.575464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:25.575429 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:16:25.575527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:25.575500 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:16:25.575581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:25.575552 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:16:26.031152 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:26.031121 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:26.130788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:26.130753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:26.481250 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:26.466047 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:26.592278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:26.592239 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" event={"ID":"555966c0-f453-4441-8991-cf25c1200236","Type":"ContainerStarted","Data":"cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a"} Nov 04 00:16:26.592927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:26.592892 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" event={"ID":"555966c0-f453-4441-8991-cf25c1200236","Type":"ContainerStarted","Data":"d16a192095da4c8b5cc0bb61d59ccfbaf03a506b130a82a15ee0e06211e911dd"} Nov 04 00:16:27.051808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:27.051771 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:27.114057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:27.108872 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:27.159147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:27.158413 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:27.161616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:27.161226 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:27.190368 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:27.190313 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podStartSLOduration=3.190298901 podStartE2EDuration="3.190298901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:26.622308487 +0000 UTC m=+2805.444904650" watchObservedRunningTime="2026-11-04 00:16:27.190298901 +0000 UTC m=+2806.012895143" Nov 04 00:16:27.330303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:27.330210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:27.432771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:27.432734 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:27.642131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:27.638955 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:27.661190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:27.647304 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:16:28.146987 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:28.146952 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:28.937065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:28.936993 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:29.813900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:29.813860 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:30.648072 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:30.648027 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:31.412549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:31.412505 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:32.294936 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.294900 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m"] Nov 04 00:16:32.299980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.299951 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.309955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.309926 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m"] Nov 04 00:16:32.314484 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.314459 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8cc1c0-serving-cert\"" Nov 04 00:16:32.342402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.342369 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:32.423470 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.423429 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.423687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.423666 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-maas-api-tls\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.423810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.423789 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqlh\" (UniqueName: \"kubernetes.io/projected/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-kube-api-access-9wqlh\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.423850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.423823 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-metrics-certs\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.442210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.442178 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:32.532402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.526175 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-metrics-certs\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.532402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.526310 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.532402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.526558 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-maas-api-tls\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.532402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.526689 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqlh\" (UniqueName: \"kubernetes.io/projected/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-kube-api-access-9wqlh\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.533840 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.533793 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.551077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.550144 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-maas-api-tls\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.551077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.550708 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-metrics-certs\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.552063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.551960 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqlh\" (UniqueName: \"kubernetes.io/projected/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-kube-api-access-9wqlh\") pod \"maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:32.629895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:32.629857 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:33.052091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:33.045205 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m"] Nov 04 00:16:33.063336 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:33.063313 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:16:33.463129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:33.463075 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:33.772759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:33.772672 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" event={"ID":"71e0e69d-ea9d-4bf1-b4cb-14c10731a974","Type":"ContainerStarted","Data":"ca90f5410c1a6f9b730b287e59cfa0a6e119996096d9fb771586fad20022c396"} Nov 04 00:16:34.205409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:34.205375 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:34.782075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:34.782013 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" event={"ID":"71e0e69d-ea9d-4bf1-b4cb-14c10731a974","Type":"ContainerStarted","Data":"0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5"} Nov 04 00:16:34.782525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:34.782181 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:34.804537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:34.804490 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" podStartSLOduration=2.174935475 podStartE2EDuration="2.804477863s" totalImagesPullingTime="629.542388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:16:34.802641659 +0000 UTC m=+2813.625237826" watchObservedRunningTime="2026-11-04 00:16:34.804477863 +0000 UTC m=+2813.627074025" Nov 04 00:16:34.943484 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:34.943387 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:35.712466 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:16:35.712417 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:16:35.721774 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:35.721748 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:36.449655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:36.449619 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:37.225987 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:37.225951 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:37.681888 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:37.681811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:38.120728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:38.120688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:38.902253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:38.902217 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:39.009012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:39.008978 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:39.108901 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:39.108869 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:40.003625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:40.003569 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:40.239661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:40.239627 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:40.339084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:40.338957 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:40.837572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:40.837542 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:16:40.939385 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:40.939295 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:41.852023 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:41.851987 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:42.551863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:42.551829 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:43.255411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:43.255370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:43.995517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:43.995480 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:44.844961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:44.844924 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:45.564299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:45.564265 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:46.326256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:46.326223 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:47.326097 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:47.326029 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:48.142658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:48.142624 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:49.008848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:49.008804 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:49.121061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:49.116312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:49.833893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:49.833854 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:50.650853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:50.650804 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:51.494773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:51.494733 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:52.224125 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:52.224090 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:53.045572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:53.045534 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:53.831085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:53.831026 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:54.556178 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:54.556134 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:55.447600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:55.447567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:56.155373 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:56.155338 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:57.076021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:57.075970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:58.021127 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:58.021089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:58.825641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:58.825604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:16:59.547377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:16:59.547339 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:00.343772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:00.343732 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:01.042096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.042060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:01.769600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.769564 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6"] Nov 04 00:17:01.774501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.774475 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.780064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.780020 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-954cd9-serving-cert\"" Nov 04 00:17:01.787106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.787074 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6"] Nov 04 00:17:01.868093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.868019 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:01.889014 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.888983 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-maas-api-tls\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.889193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.889134 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-metrics-certs\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.889193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.889168 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d8c34f-fd93-4700-8430-f75fcc6a670f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.889280 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.889248 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxfh\" (UniqueName: \"kubernetes.io/projected/46d8c34f-fd93-4700-8430-f75fcc6a670f-kube-api-access-rjxfh\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.992888 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.990694 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-maas-api-tls\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.992888 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.990829 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-metrics-certs\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.992888 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.990867 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d8c34f-fd93-4700-8430-f75fcc6a670f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.992888 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.990940 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxfh\" (UniqueName: \"kubernetes.io/projected/46d8c34f-fd93-4700-8430-f75fcc6a670f-kube-api-access-rjxfh\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.998502 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.997905 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-maas-api-tls\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.998993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.998944 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d8c34f-fd93-4700-8430-f75fcc6a670f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:01.999518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:01.999453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:02.004963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:02.004905 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-metrics-certs\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:02.008352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:02.008295 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxfh\" (UniqueName: \"kubernetes.io/projected/46d8c34f-fd93-4700-8430-f75fcc6a670f-kube-api-access-rjxfh\") pod \"maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:02.108535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:02.108449 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:02.399098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:02.399065 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6"] Nov 04 00:17:02.400116 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:17:02.400083 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d8c34f_fd93_4700_8430_f75fcc6a670f.slice/crio-e3e09957c21c659de31c8285cb2a7e36e19ad9e27e5dfe9e0ef53d00b7714339 WatchSource:0}: Error finding container e3e09957c21c659de31c8285cb2a7e36e19ad9e27e5dfe9e0ef53d00b7714339: Status 404 returned error can't find the container with id e3e09957c21c659de31c8285cb2a7e36e19ad9e27e5dfe9e0ef53d00b7714339 Nov 04 00:17:02.790242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:02.790211 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:03.252962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:03.252920 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" event={"ID":"46d8c34f-fd93-4700-8430-f75fcc6a670f","Type":"ContainerStarted","Data":"5fbfcdf72d82ba7dd7de926a985b191059d3eaaaaf022bbdead056d8f48f7b63"} Nov 04 00:17:03.253464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:03.252976 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" event={"ID":"46d8c34f-fd93-4700-8430-f75fcc6a670f","Type":"ContainerStarted","Data":"e3e09957c21c659de31c8285cb2a7e36e19ad9e27e5dfe9e0ef53d00b7714339"} Nov 04 00:17:03.253464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:03.253007 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:03.277156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:03.277087 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" podStartSLOduration=1.6010351470000002 podStartE2EDuration="2.277069604s" totalImagesPullingTime="676.034457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:03.273503177 +0000 UTC m=+2842.096099340" watchObservedRunningTime="2026-11-04 00:17:03.277069604 +0000 UTC m=+2842.099665764" Nov 04 00:17:03.525169 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:03.525132 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:04.272177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:04.272142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:05.036535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:05.036498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:05.709685 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:17:05.709649 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:17:05.748079 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:05.748049 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:06.628210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:06.628162 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:07.336439 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:07.336400 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:08.043267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:08.043229 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:08.422461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:08.422363 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:08.946318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:08.946285 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:09.267027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:09.266993 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:17:09.802589 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:09.802551 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:10.501854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:10.501817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:10.628874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:10.628841 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:11.528234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:11.524889 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:12.228514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:12.228478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:13.045490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:13.045456 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:14.025139 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:14.025106 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:14.844027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:14.843989 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:15.592330 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:15.592288 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:16.326684 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:16.326643 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:16.679331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:16.679235 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:17.203018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.202982 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:17.307782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.307743 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:17.640863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.640822 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq"] Nov 04 00:17:17.645466 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.645441 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.653154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.653125 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:17.658292 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.657631 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq"] Nov 04 00:17:17.682924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.682895 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-85ad28-openshift-default-dockercfg-5xfzh\"" Nov 04 00:17:17.764248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.764204 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.871573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.871540 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.871908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.871875 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.872239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.872213 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.872404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.872386 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7k7\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-kube-api-access-5l7k7\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.872561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.872545 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.872740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.872725 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.872955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.872938 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.873157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.873125 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.873403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.868783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:17.976153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.976106 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.976518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.976494 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.976655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.976640 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7k7\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-kube-api-access-5l7k7\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.976770 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.976757 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.976917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.976904 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.995057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.977089 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.995512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.995483 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.995737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.995715 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.995915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.995896 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.996542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.993887 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.996712 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.995196 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.997111 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.994289 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:17.997879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:17.997856 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:18.001959 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.001898 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:18.082086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.082023 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:18.082410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.082383 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:18.085342 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.085277 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7k7\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-kube-api-access-5l7k7\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:18.104697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.104665 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:18.248076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.247957 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:18.266391 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.266353 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:18.627461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.627432 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq"] Nov 04 00:17:18.628070 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:17:18.628018 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb44f32_c801_4a3b_b97b_81e776ddbbd6.slice/crio-760ea9bd4b504487fcf58fd5209a631e48883530064ee69df28e43e6d6629027 WatchSource:0}: Error finding container 760ea9bd4b504487fcf58fd5209a631e48883530064ee69df28e43e6d6629027: Status 404 returned error can't find the container with id 760ea9bd4b504487fcf58fd5209a631e48883530064ee69df28e43e6d6629027 Nov 04 00:17:18.638136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.638093 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:17:18.638225 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.638175 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:17:18.638225 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:18.638205 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:17:19.057529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:19.057493 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:19.392331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:19.392238 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" event={"ID":"3fb44f32-c801-4a3b-b97b-81e776ddbbd6","Type":"ContainerStarted","Data":"3eb4f86e887ae23511d74fd2b3666eddc86fcf3c7f45618ee9de9a605f878654"} Nov 04 00:17:19.392331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:19.392307 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" event={"ID":"3fb44f32-c801-4a3b-b97b-81e776ddbbd6","Type":"ContainerStarted","Data":"760ea9bd4b504487fcf58fd5209a631e48883530064ee69df28e43e6d6629027"} Nov 04 00:17:19.419460 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:19.419399 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podStartSLOduration=2.419384581 podStartE2EDuration="2.419384581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:17:19.416298238 +0000 UTC m=+2858.238894401" watchObservedRunningTime="2026-11-04 00:17:19.419384581 +0000 UTC m=+2858.241980742" Nov 04 00:17:20.041183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:20.041144 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:20.267369 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:20.267205 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:20.274207 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:20.274172 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:20.285618 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:20.285583 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:20.399993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:20.399878 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:20.463057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:20.450750 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:17:21.171988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:21.171920 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:21.932828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:21.932772 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:22.042641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:22.042602 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:22.219612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:22.219576 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:23.322421 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:23.322382 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:24.014984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:24.014937 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:24.127254 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:24.127216 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:24.399582 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:24.399489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:25.227419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:25.227378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:26.205571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:26.205537 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:26.521245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:26.521164 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:27.034388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:27.034350 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:27.742897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:27.742867 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:28.570833 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:28.570797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:29.429604 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:29.429564 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:29.985203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:29.985162 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:30.400945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:30.400854 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:30.531793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:30.531679 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:31.000344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:31.000297 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:31.231059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:31.227471 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:31.546556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:31.546505 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:32.341561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:32.341522 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:33.238698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:33.238662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:34.087924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:34.087881 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:34.940254 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:34.940210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:35.117443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:35.117408 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:35.229886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:35.229841 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:35.821691 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:35.821654 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:36.198090 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:17:36.197940 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:17:36.367708 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:36.367611 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:36.471884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:36.471846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:36.952091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:36.951992 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:37.740811 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:37.740778 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:37.921554 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:37.921513 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:38.739147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:38.739113 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:39.485609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:39.485578 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:40.290141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:40.290095 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:41.225025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:41.224990 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:41.934744 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:41.934705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:42.744861 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:42.744686 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:43.622356 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:43.622320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:43.738238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:43.737766 6208 patch_prober.go:29] interesting pod/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:17:43.738238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:43.737865 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:17:44.330180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:44.330139 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:45.123093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:45.123061 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:45.906025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:45.905982 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:45.911769 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:45.911740 6208 generic.go:356] "Generic (PLEG): container finished" podID="555966c0-f453-4441-8991-cf25c1200236" containerID="cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a" exitCode=0 Nov 04 00:17:45.911940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:45.911820 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" event={"ID":"555966c0-f453-4441-8991-cf25c1200236","Type":"ContainerDied","Data":"cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a"} Nov 04 00:17:45.912381 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:45.912360 6208 scope.go:122] "RemoveContainer" containerID="cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a" Nov 04 00:17:45.914522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:45.914490 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:17:45.914629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:45.914573 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:17:45.914629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:45.914610 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:17:46.625290 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:46.625249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:46.922374 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:46.922285 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" event={"ID":"555966c0-f453-4441-8991-cf25c1200236","Type":"ContainerStarted","Data":"b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846"} Nov 04 00:17:47.104790 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:47.104753 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:17:47.105006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:47.104969 6208 patch_prober.go:29] interesting pod/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 00:17:47.105108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:47.105049 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 00:17:47.346396 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:47.346357 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:47.445867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:47.445829 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:48.105944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:48.105904 6208 patch_prober.go:29] interesting pod/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 00:17:48.106522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:48.105970 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 00:17:48.223286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:48.223249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:49.105643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:49.104965 6208 patch_prober.go:29] interesting pod/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 00:17:49.105643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:49.105028 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 00:17:49.231584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:49.231544 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:50.105728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:50.105640 6208 patch_prober.go:29] interesting pod/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 00:17:50.105728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:50.105699 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 00:17:50.177559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:50.177521 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:50.278494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:50.278378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:51.117136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:51.115523 6208 patch_prober.go:29] interesting pod/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 00:17:51.117136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:51.115581 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 00:17:51.182781 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:51.182744 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:51.380558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:51.380449 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:52.105901 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:52.105861 6208 patch_prober.go:29] interesting pod/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 00:17:52.106102 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:52.105916 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 00:17:52.135163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:52.135125 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:52.430926 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:52.430834 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7"] Nov 04 00:17:52.489369 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:52.489336 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:52.589209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:52.589178 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:52.976067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:52.976012 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8"] Nov 04 00:17:53.023881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:53.023846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:53.137100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:53.137029 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:53.398650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:53.398538 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:53.497175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:53.497141 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:53.570740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:53.570711 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:17:53.571368 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:53.571326 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:17:53.572661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:53.572633 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:17:54.132468 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:54.132428 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:54.917951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:54.917915 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:55.723439 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:55.723397 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:56.436273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:56.436238 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:57.252629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.252590 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:57.431302 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.431261 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" podUID="bcc4333e-ca3e-40be-931c-1068b60d2915" containerName="payload-processing" containerID="cri-o://608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa" gracePeriod=25 Nov 04 00:17:57.823194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.823168 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:17:57.873662 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.873613 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-plugins-config-volume\") pod \"bcc4333e-ca3e-40be-931c-1068b60d2915\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " Nov 04 00:17:57.873847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.873717 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-kube-api-access-8xjnx\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-kube-api-access-8xjnx\") pod \"bcc4333e-ca3e-40be-931c-1068b60d2915\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " Nov 04 00:17:57.873940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.873923 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-routing-overlay-volume\") pod \"bcc4333e-ca3e-40be-931c-1068b60d2915\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " Nov 04 00:17:57.874831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.874108 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-plugins-config-volume" pod "bcc4333e-ca3e-40be-931c-1068b60d2915" (UID: "bcc4333e-ca3e-40be-931c-1068b60d2915"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:17:57.874831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.874134 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-provider-credentials\") pod \"bcc4333e-ca3e-40be-931c-1068b60d2915\" (UID: \"bcc4333e-ca3e-40be-931c-1068b60d2915\") " Nov 04 00:17:57.874831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.874360 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-routing-overlay-volume" pod "bcc4333e-ca3e-40be-931c-1068b60d2915" (UID: "bcc4333e-ca3e-40be-931c-1068b60d2915"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:17:57.874957 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.874915 6208 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-routing-overlay-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:17:57.874957 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.874941 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc4333e-ca3e-40be-931c-1068b60d2915-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:17:57.878006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.877974 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-kube-api-access-8xjnx" pod "bcc4333e-ca3e-40be-931c-1068b60d2915" (UID: "bcc4333e-ca3e-40be-931c-1068b60d2915"). InnerVolumeSpecName "kube-api-access-8xjnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:17:57.878141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.878015 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-provider-credentials" pod "bcc4333e-ca3e-40be-931c-1068b60d2915" (UID: "bcc4333e-ca3e-40be-931c-1068b60d2915"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:17:57.976546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.976506 6208 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-provider-credentials\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:17:57.976546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.976544 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xjnx\" (UniqueName: \"kubernetes.io/projected/bcc4333e-ca3e-40be-931c-1068b60d2915-kube-api-access-8xjnx\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:17:57.976766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:57.976572 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" podUID="f229cbce-2905-47a2-af7d-33750f50ee81" containerName="payload-processing" containerID="cri-o://64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851" gracePeriod=25 Nov 04 00:17:58.035273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.035233 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:58.160526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.129755 6208 generic.go:356] "Generic (PLEG): container finished" podID="bcc4333e-ca3e-40be-931c-1068b60d2915" containerID="608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa" exitCode=0 Nov 04 00:17:58.160526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.129943 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" event={"ID":"bcc4333e-ca3e-40be-931c-1068b60d2915","Type":"ContainerDied","Data":"608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa"} Nov 04 00:17:58.160526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.129995 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" event={"ID":"bcc4333e-ca3e-40be-931c-1068b60d2915","Type":"ContainerDied","Data":"7573422b76995cc6782472e5159e9d311d875642340618dce22d3645dae4ed82"} Nov 04 00:17:58.160526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.130022 6208 scope.go:122] "RemoveContainer" containerID="608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa" Nov 04 00:17:58.160526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.130218 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7" Nov 04 00:17:58.174282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.174231 6208 scope.go:122] "RemoveContainer" containerID="608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa" Nov 04 00:17:58.174625 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:17:58.174603 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa\": container with ID starting with 608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa not found: ID does not exist" containerID="608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa" Nov 04 00:17:58.174690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.174636 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa"} err="failed to get container status \"608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa\": rpc error: code = NotFound desc = could not find container \"608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa\": container with ID starting with 608ce2ed9618800c2b89f41fa3f9e6cef9d2a7c13ef4bde409b875914b796ffa not found: ID does not exist" Nov 04 00:17:58.180657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.180622 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7"] Nov 04 00:17:58.185211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.185188 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5fca/payload-processing-external-model-e2e-extmod-b-1b5fca-6fd4xblk7"] Nov 04 00:17:58.319733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.319702 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:17:58.386922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.386894 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-provider-credentials\") pod \"f229cbce-2905-47a2-af7d-33750f50ee81\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " Nov 04 00:17:58.387113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.387012 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-routing-overlay-volume\") pod \"f229cbce-2905-47a2-af7d-33750f50ee81\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " Nov 04 00:17:58.387173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.387162 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-plugins-config-volume\") pod \"f229cbce-2905-47a2-af7d-33750f50ee81\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " Nov 04 00:17:58.387277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.387260 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-kube-api-access-k4msc\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-kube-api-access-k4msc\") pod \"f229cbce-2905-47a2-af7d-33750f50ee81\" (UID: \"f229cbce-2905-47a2-af7d-33750f50ee81\") " Nov 04 00:17:58.387693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.387516 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-routing-overlay-volume" pod "f229cbce-2905-47a2-af7d-33750f50ee81" (UID: "f229cbce-2905-47a2-af7d-33750f50ee81"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:17:58.387693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.387619 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-plugins-config-volume" pod "f229cbce-2905-47a2-af7d-33750f50ee81" (UID: "f229cbce-2905-47a2-af7d-33750f50ee81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:17:58.388643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.388617 6208 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-routing-overlay-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:17:58.388771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.388645 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f229cbce-2905-47a2-af7d-33750f50ee81-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:17:58.389733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.389710 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-kube-api-access-k4msc" pod "f229cbce-2905-47a2-af7d-33750f50ee81" (UID: "f229cbce-2905-47a2-af7d-33750f50ee81"). InnerVolumeSpecName "kube-api-access-k4msc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:17:58.389891 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.389740 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-provider-credentials" pod "f229cbce-2905-47a2-af7d-33750f50ee81" (UID: "f229cbce-2905-47a2-af7d-33750f50ee81"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:17:58.490279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.490246 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4msc\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-kube-api-access-k4msc\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:17:58.490279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.490275 6208 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f229cbce-2905-47a2-af7d-33750f50ee81-provider-credentials\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:17:58.920831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:58.920735 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:17:59.139706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.139672 6208 generic.go:356] "Generic (PLEG): container finished" podID="f229cbce-2905-47a2-af7d-33750f50ee81" containerID="64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851" exitCode=0 Nov 04 00:17:59.139889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.139727 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" Nov 04 00:17:59.139889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.139760 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" event={"ID":"f229cbce-2905-47a2-af7d-33750f50ee81","Type":"ContainerDied","Data":"64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851"} Nov 04 00:17:59.139889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.139816 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8" event={"ID":"f229cbce-2905-47a2-af7d-33750f50ee81","Type":"ContainerDied","Data":"e29829a9984336ef492d775c3b06aac9c87cc9f028605c29964b2081dcb515e2"} Nov 04 00:17:59.139889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.139841 6208 scope.go:122] "RemoveContainer" containerID="64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851" Nov 04 00:17:59.152531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.152509 6208 scope.go:122] "RemoveContainer" containerID="64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851" Nov 04 00:17:59.152796 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:17:59.152775 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851\": container with ID starting with 64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851 not found: ID does not exist" containerID="64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851" Nov 04 00:17:59.152870 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.152810 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851"} err="failed to get container status \"64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851\": rpc error: code = NotFound desc = could not find container \"64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851\": container with ID starting with 64e2010d26172101b3a53bc66a207b6994cfd00cb19b6c73cb82b7065dafb851 not found: ID does not exist" Nov 04 00:17:59.176396 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.176304 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8"] Nov 04 00:17:59.178818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.178788 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a002b/payload-processing-external-model-e2e-extmod-a-3a002b-57dddfdq8"] Nov 04 00:17:59.295734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.295701 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc4333e-ca3e-40be-931c-1068b60d2915" path="/var/lib/kubelet/pods/bcc4333e-ca3e-40be-931c-1068b60d2915/volumes" Nov 04 00:17:59.296216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.296202 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f229cbce-2905-47a2-af7d-33750f50ee81" path="/var/lib/kubelet/pods/f229cbce-2905-47a2-af7d-33750f50ee81/volumes" Nov 04 00:17:59.772824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:17:59.772780 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:00.115540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.115445 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:00.223180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.220568 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:00.618613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.618579 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:00.658404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.658368 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt"] Nov 04 00:18:00.659437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.659419 6208 cpu_manager.go:414] "containerMap: removing container" podUID="bcc4333e-ca3e-40be-931c-1068b60d2915" containerName="payload-processing" Nov 04 00:18:00.659539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.659439 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc4333e-ca3e-40be-931c-1068b60d2915" containerName="payload-processing" Nov 04 00:18:00.659539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.659470 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f229cbce-2905-47a2-af7d-33750f50ee81" containerName="payload-processing" Nov 04 00:18:00.659539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.659478 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f229cbce-2905-47a2-af7d-33750f50ee81" containerName="payload-processing" Nov 04 00:18:00.660392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.659649 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc4333e-ca3e-40be-931c-1068b60d2915" containerName="payload-processing" Nov 04 00:18:00.660392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.659665 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f229cbce-2905-47a2-af7d-33750f50ee81" containerName="payload-processing" Nov 04 00:18:00.665210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.665187 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.679313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.677290 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a7cbc3-serving-cert\"" Nov 04 00:18:00.682197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.682163 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt"] Nov 04 00:18:00.720084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.720047 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:00.730108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.730074 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pclnk\" (UniqueName: \"kubernetes.io/projected/28b0b70f-ad5e-47e5-af92-f743b51805ad-kube-api-access-pclnk\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.730331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.730238 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0b70f-ad5e-47e5-af92-f743b51805ad-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.730331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.730279 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-metrics-certs\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.731211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.730343 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.843055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.833797 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclnk\" (UniqueName: \"kubernetes.io/projected/28b0b70f-ad5e-47e5-af92-f743b51805ad-kube-api-access-pclnk\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.843055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.834075 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0b70f-ad5e-47e5-af92-f743b51805ad-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.843055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.834134 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-metrics-certs\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.843055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.834181 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.843055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.835996 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:00.843055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.836802 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0b70f-ad5e-47e5-af92-f743b51805ad-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.843055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.841303 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-metrics-certs\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.851874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.849938 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.860209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.860164 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pclnk\" (UniqueName: \"kubernetes.io/projected/28b0b70f-ad5e-47e5-af92-f743b51805ad-kube-api-access-pclnk\") pod \"maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:00.943020 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.942928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:00.995120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:00.995083 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:01.022217 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.022174 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n"] Nov 04 00:18:01.028179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.028150 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.032693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.032277 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d94e94-openshift-default-dockercfg-dfzz6\"" Nov 04 00:18:01.038751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.037585 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n"] Nov 04 00:18:01.045764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.045735 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:01.139790 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.139709 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-certs\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.139790 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.139773 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-podinfo\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.140404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.139876 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-envoy\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.140404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.139962 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-data\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.140404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.140022 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-credential-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.140404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.140106 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1903fe-748c-4b83-9d58-7727b1f9b755-istiod-ca-cert\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.140404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.140189 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.140404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.140316 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-token\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.140739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.140409 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkfrc\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-kube-api-access-dkfrc\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.144980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.144949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:01.242444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.242398 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1903fe-748c-4b83-9d58-7727b1f9b755-istiod-ca-cert\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.242623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.242502 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.242713 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.242629 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-token\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.242781 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.242720 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkfrc\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-kube-api-access-dkfrc\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.243064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.242906 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.243402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.243213 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1903fe-748c-4b83-9d58-7727b1f9b755-istiod-ca-cert\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.243533 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.243506 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-certs\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.243592 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.243552 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-podinfo\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.243592 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.243580 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-envoy\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.244282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.243649 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-data\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.244282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.243700 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-credential-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.244282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.243800 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-certs\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.244282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.244091 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-credential-socket\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.244910 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.244336 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-data\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.248027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.246643 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-envoy\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.248027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.247157 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-podinfo\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.277418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.277315 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkfrc\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-kube-api-access-dkfrc\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.326090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.296644 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-token\") pod \"e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.419351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.414342 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:01.479104 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:18:01.478188 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b0b70f_ad5e_47e5_af92_f743b51805ad.slice/crio-83de62c7dbda8403e0a3f928fdc4fbceccf43b81e1557884361daf2e20782be6 WatchSource:0}: Error finding container 83de62c7dbda8403e0a3f928fdc4fbceccf43b81e1557884361daf2e20782be6: Status 404 returned error can't find the container with id 83de62c7dbda8403e0a3f928fdc4fbceccf43b81e1557884361daf2e20782be6 Nov 04 00:18:01.533960 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.533930 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt"] Nov 04 00:18:01.727160 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.727107 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:01.899020 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.898992 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n"] Nov 04 00:18:01.915155 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:18:01.910157 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1903fe_748c_4b83_9d58_7727b1f9b755.slice/crio-e909317c8daf5981c491554a9eae67cdb6e9fbc5a021ed1b4d57f13e24abb585 WatchSource:0}: Error finding container e909317c8daf5981c491554a9eae67cdb6e9fbc5a021ed1b4d57f13e24abb585: Status 404 returned error can't find the container with id e909317c8daf5981c491554a9eae67cdb6e9fbc5a021ed1b4d57f13e24abb585 Nov 04 00:18:01.915155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.914676 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:01.915155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.914831 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:01.915155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:01.915006 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:02.170745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:02.170654 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" event={"ID":"28b0b70f-ad5e-47e5-af92-f743b51805ad","Type":"ContainerStarted","Data":"8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b"} Nov 04 00:18:02.171230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:02.170757 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" event={"ID":"28b0b70f-ad5e-47e5-af92-f743b51805ad","Type":"ContainerStarted","Data":"83de62c7dbda8403e0a3f928fdc4fbceccf43b81e1557884361daf2e20782be6"} Nov 04 00:18:02.171230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:02.170963 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:02.172659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:02.172631 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" event={"ID":"1b1903fe-748c-4b83-9d58-7727b1f9b755","Type":"ContainerStarted","Data":"72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2"} Nov 04 00:18:02.173384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:02.172675 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" event={"ID":"1b1903fe-748c-4b83-9d58-7727b1f9b755","Type":"ContainerStarted","Data":"e909317c8daf5981c491554a9eae67cdb6e9fbc5a021ed1b4d57f13e24abb585"} Nov 04 00:18:02.192832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:02.192777 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" podStartSLOduration=1.6642027719999999 podStartE2EDuration="2.192762216s" totalImagesPullingTime="528.559444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:02.190014454 +0000 UTC m=+2901.012610620" watchObservedRunningTime="2026-11-04 00:18:02.192762216 +0000 UTC m=+2901.015358377" Nov 04 00:18:02.215250 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:02.215200 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" podStartSLOduration=2.215186005 podStartE2EDuration="2.215186005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:02.212084575 +0000 UTC m=+2901.034680737" watchObservedRunningTime="2026-11-04 00:18:02.215186005 +0000 UTC m=+2901.037782166" Nov 04 00:18:02.416225 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:02.416185 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:02.681550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:02.681501 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:03.423825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:03.423793 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:03.493559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:03.493516 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:04.190181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:04.190148 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:04.191428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:04.191406 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:18:04.360681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:04.360642 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:05.843062 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:18:05.841356 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:18:06.012772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:06.012601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:06.937065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:06.936997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:07.931127 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:07.931021 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:08.205107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:08.204633 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:18:08.814059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:08.814018 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:09.237404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:09.236509 6208 generic.go:356] "Generic (PLEG): container finished" podID="c9bf53bb-bfda-4935-8536-dd79e6e7c6c7" containerID="5ab4469b3f7e503fd3bb2dc653ed559796e9cdf7d1f6992e94f58e43b521aa3f" exitCode=1 Nov 04 00:18:09.237404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:09.236653 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" event={"ID":"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7","Type":"ContainerDied","Data":"5ab4469b3f7e503fd3bb2dc653ed559796e9cdf7d1f6992e94f58e43b521aa3f"} Nov 04 00:18:09.237404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:09.237209 6208 scope.go:122] "RemoveContainer" containerID="5ab4469b3f7e503fd3bb2dc653ed559796e9cdf7d1f6992e94f58e43b521aa3f" Nov 04 00:18:09.240589 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:09.240563 6208 generic.go:356] "Generic (PLEG): container finished" podID="07f6fefb-f056-4495-a778-187e86acc017" containerID="d329ea3624a5804bc05aa795197fd6cd78f857707c53b989a06120a768ffe716" exitCode=1 Nov 04 00:18:09.240716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:09.240640 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" event={"ID":"07f6fefb-f056-4495-a778-187e86acc017","Type":"ContainerDied","Data":"d329ea3624a5804bc05aa795197fd6cd78f857707c53b989a06120a768ffe716"} Nov 04 00:18:09.241049 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:09.241018 6208 scope.go:122] "RemoveContainer" containerID="d329ea3624a5804bc05aa795197fd6cd78f857707c53b989a06120a768ffe716" Nov 04 00:18:09.535165 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:09.534968 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:09.924428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:09.924397 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 04 00:18:10.252053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:10.252008 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" event={"ID":"c9bf53bb-bfda-4935-8536-dd79e6e7c6c7","Type":"ContainerStarted","Data":"c0e8fe264ac553560ebf0cf4dbef708f283eb1921bc28fbac007ee1a6c173ef0"} Nov 04 00:18:10.252551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:10.252378 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 04 00:18:10.274053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:10.264438 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" event={"ID":"07f6fefb-f056-4495-a778-187e86acc017","Type":"ContainerStarted","Data":"28a3aca042e5aa538ccf55b44886c8ced8d164e0c05a84f6d6973d8c381f2182"} Nov 04 00:18:10.274053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:10.269461 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 04 00:18:10.336639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:10.336584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:11.117520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:11.117478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:11.829396 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:11.829346 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:12.548729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:12.548688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:13.307989 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:13.307950 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:14.050206 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:14.050165 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:14.837610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:14.837572 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:15.537723 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:15.537687 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:16.338686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:16.338649 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:17.058591 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:17.058549 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:17.841004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:17.840971 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:18.064371 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:18.064335 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:18.659879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:18.659838 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:19.638643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:19.638604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:20.440568 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:20.440530 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:20.487888 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:20.461853 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-c5hhg" podUID="63ac6df1-998d-43e6-bf49-2aeff97e4a4b" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.25:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:18:20.740280 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:20.738006 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 00:18:20.799619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:20.766631 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fj7th" podUID="ef180863-1732-423f-a932-989ba01cb99c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 00:18:20.799619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:20.775604 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv"] Nov 04 00:18:21.152933 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.152846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:21.196528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.196490 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv"] Nov 04 00:18:21.196699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.196586 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.253794 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.253745 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.296402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.253934 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d797191-b9e1-4fc7-88bb-4eda431424b4-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.296402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.254089 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.296402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.254163 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.296402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.254214 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.296402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.254355 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.296402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.254416 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.296402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.254550 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.296402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.254597 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22k97\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-kube-api-access-22k97\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.296402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.263484 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.355780 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d797191-b9e1-4fc7-88bb-4eda431424b4-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.355869 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.355921 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.355971 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.356051 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.356103 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.356175 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.356209 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22k97\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-kube-api-access-22k97\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.356258 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.441210 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:18:21.492277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.441276 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:18:21.756859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.641201 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-74c78cbdbf-gxkf7" Nov 04 00:18:21.756859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.647301 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.822235 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.822592 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d797191-b9e1-4fc7-88bb-4eda431424b4-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.822728 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:21.822920 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:22.085214 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:22.085284 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:22.364202 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" podUID="c9bf53bb-bfda-4935-8536-dd79e6e7c6c7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:22.367552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:22.437384 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:22.464508 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:22.493508 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22k97\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-kube-api-access-22k97\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:22.527496 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:22.805013 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:23.980851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:23.248337 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:24.267255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:24.151251 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:25.106635 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:25.106600 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:26.082179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:26.029788 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:26.330990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:26.330958 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv"] Nov 04 00:18:26.377098 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:18:26.377061 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d797191_b9e1_4fc7_88bb_4eda431424b4.slice/crio-252e8bb6c272207752281416088f877dfda643fefea0745ddb7911ee40114af1 WatchSource:0}: Error finding container 252e8bb6c272207752281416088f877dfda643fefea0745ddb7911ee40114af1: Status 404 returned error can't find the container with id 252e8bb6c272207752281416088f877dfda643fefea0745ddb7911ee40114af1 Nov 04 00:18:26.391434 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:26.391397 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:26.391525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:26.391481 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:26.391525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:26.391511 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:26.698523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:26.698488 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:27.046406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:27.046369 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:27.060326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:27.052989 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" event={"ID":"2d797191-b9e1-4fc7-88bb-4eda431424b4","Type":"ContainerStarted","Data":"49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1"} Nov 04 00:18:27.060326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:27.053062 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" event={"ID":"2d797191-b9e1-4fc7-88bb-4eda431424b4","Type":"ContainerStarted","Data":"252e8bb6c272207752281416088f877dfda643fefea0745ddb7911ee40114af1"} Nov 04 00:18:27.090884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:27.087899 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podStartSLOduration=9.087876711 podStartE2EDuration="9.087876711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:27.080709084 +0000 UTC m=+2925.903305245" watchObservedRunningTime="2026-11-04 00:18:27.087876711 +0000 UTC m=+2925.910472873" Nov 04 00:18:27.806050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:27.805573 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:27.808060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:27.807707 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:27.808060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:27.807797 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:27.921021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:27.920977 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:28.726513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:28.726479 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:28.822060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:28.818018 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:28.822060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:28.818098 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:29.617842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:29.617355 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:29.833059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:29.830453 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:29.833059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:29.830529 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:30.574488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:30.574436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:30.805565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:30.805530 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:30.805853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:30.805829 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:31.205678 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:31.205639 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:31.332980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:31.332943 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6db599696d-4njr8" Nov 04 00:18:31.402222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:31.402189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:31.519046 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:31.518903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:31.698911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:31.698772 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:31.806641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:31.806520 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:31.806951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:31.806902 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:31.943605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:31.943561 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:32.486589 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.486547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:32.794181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.793544 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768"] Nov 04 00:18:32.809058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.807293 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:32.809058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.807351 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:32.823439 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.822460 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.832299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.832258 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768"] Nov 04 00:18:32.883440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.883172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:32.911026 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.910989 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.913541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.911109 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.913541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.911142 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.913541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.911191 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.913541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.911226 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k82wp\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-kube-api-access-k82wp\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.913541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.911245 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.913541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.911332 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.913541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.911355 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.913541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.911408 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:32.991763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:32.991523 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:33.012716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.012670 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.012877 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.012733 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.012877 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.012824 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.013005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.012964 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.013100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.013083 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.013154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.013117 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.013210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.013167 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.020639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.013211 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k82wp\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-kube-api-access-k82wp\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.020639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.013237 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.035529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.035493 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.038141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.037807 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.038141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.037983 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.038141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.038107 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.038484 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.038382 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.040105 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.040077 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k82wp\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-kube-api-access-k82wp\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.042816 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.042793 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.108951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.108854 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.110073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.109998 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.200083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.199795 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:33.717584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.717410 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:33.877084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.872679 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:33.877084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.872740 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:33.934237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:33.934132 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b"] Nov 04 00:18:34.013433 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.013395 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b"] Nov 04 00:18:34.013684 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.013494 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.037797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.037760 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4526c8ef-a23a-492b-99cf-31957ea71f0f-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.038082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.037841 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.038082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.037940 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz2rq\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-kube-api-access-zz2rq\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.038082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.037993 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.038444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.038146 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.038444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.038181 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.038444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.038216 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.038444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.038272 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.038444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.038390 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.106769 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.100095 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768"] Nov 04 00:18:34.116157 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:18:34.116119 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1516c89_1bc2_4251_90f8_f1d6688dfdf6.slice/crio-565527d3f3dbfee7c47536258a3a38de0b7300b12e46a699eeaf3397d3ba947e WatchSource:0}: Error finding container 565527d3f3dbfee7c47536258a3a38de0b7300b12e46a699eeaf3397d3ba947e: Status 404 returned error can't find the container with id 565527d3f3dbfee7c47536258a3a38de0b7300b12e46a699eeaf3397d3ba947e Nov 04 00:18:34.119786 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.119749 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:34.119907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.119828 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:34.119907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.119856 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:34.139621 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.139570 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4526c8ef-a23a-492b-99cf-31957ea71f0f-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.139798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.139663 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.139798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.139713 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz2rq\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-kube-api-access-zz2rq\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.139798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.139756 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.140078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.139867 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.140078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.139904 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.140078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.139938 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.140078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.139997 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.141228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.141179 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.143364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.143025 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-data\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.143364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.143321 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4526c8ef-a23a-492b-99cf-31957ea71f0f-istiod-ca-cert\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.147938 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.147885 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-certs\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.160306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.160223 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-envoy\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.193241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.166859 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.193241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.167371 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-podinfo\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.193241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.168571 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-credential-socket\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.198169 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.198132 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz2rq\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-kube-api-access-zz2rq\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.204830 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.204804 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-token\") pod \"e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.298906 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.298864 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" event={"ID":"a1516c89-1bc2-4251-90f8-f1d6688dfdf6","Type":"ContainerStarted","Data":"565527d3f3dbfee7c47536258a3a38de0b7300b12e46a699eeaf3397d3ba947e"} Nov 04 00:18:34.400012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.399972 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:34.629390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.614819 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:34.823214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.822589 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:34.823214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:34.822653 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:35.125177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.125134 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:35.126180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.125440 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:35.126180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.125493 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:35.137350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.137290 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b"] Nov 04 00:18:35.349295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.339322 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" event={"ID":"4526c8ef-a23a-492b-99cf-31957ea71f0f","Type":"ContainerStarted","Data":"75dc8e15406b51ff7417161d6c5168e217f0b1646d94b7bfd79f1d0b6150d2d9"} Nov 04 00:18:35.400480 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.399876 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" event={"ID":"a1516c89-1bc2-4251-90f8-f1d6688dfdf6","Type":"ContainerStarted","Data":"024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2"} Nov 04 00:18:35.437197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.437125 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" podStartSLOduration=3.437104082 podStartE2EDuration="3.437104082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:35.43096575 +0000 UTC m=+2934.253561951" watchObservedRunningTime="2026-11-04 00:18:35.437104082 +0000 UTC m=+2934.259700247" Nov 04 00:18:35.502065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.488959 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:35.843094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.815668 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:35.843094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:35.815730 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:36.214697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.214133 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:36.214697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.214339 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 00:18:36.214697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.214387 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 00:18:36.351861 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:18:36.351345 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:18:36.425219 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.425113 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:36.445058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.444950 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" event={"ID":"4526c8ef-a23a-492b-99cf-31957ea71f0f","Type":"ContainerStarted","Data":"fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b"} Nov 04 00:18:36.482281 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.448145 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:18:36.482281 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.448186 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:18:36.529004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.527994 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" podStartSLOduration=3.527977312 podStartE2EDuration="3.527977312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:36.513241513 +0000 UTC m=+2935.335837675" watchObservedRunningTime="2026-11-04 00:18:36.527977312 +0000 UTC m=+2935.350573474" Nov 04 00:18:36.614056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.608161 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:36.879119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.870476 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 04 00:18:36.879119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:36.870540 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 04 00:18:37.092061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.086535 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:18:37.092061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.086628 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:18:37.201607 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.201509 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 00:18:37.201951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.201912 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 00:18:37.406869 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.406274 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:37.406869 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.406418 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 00:18:37.406869 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.406461 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 00:18:37.630056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.627728 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:37.842616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.818549 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:37.918184 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:37.918070 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:38.201125 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:38.197672 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:38.205319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:38.201884 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 00:18:38.205319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:38.201953 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 00:18:38.409057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:38.408486 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 00:18:38.409057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:38.408568 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 00:18:38.523804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:38.523178 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:18:38.719065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:38.715427 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:39.211102 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:39.204414 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 00:18:39.211102 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:39.204476 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 00:18:39.402060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:39.401422 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 00:18:39.402060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:39.401490 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 00:18:39.790075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:39.787046 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:40.212803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:40.212642 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 00:18:40.212803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:40.212711 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 00:18:40.410646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:40.410483 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 00:18:40.410646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:40.410543 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 00:18:40.728921 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:40.728884 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:41.202559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:41.202467 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 04 00:18:41.213204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:41.202753 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 04 00:18:41.412961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:41.412915 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 00:18:41.413249 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:41.413001 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 00:18:41.628849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:41.628791 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:42.207743 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:42.207704 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:42.208205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:42.208179 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:42.209118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:42.209095 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:18:42.402888 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:42.402852 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 00:18:42.403833 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:42.403795 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 00:18:42.715327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:42.715290 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:43.407419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:43.407391 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:43.407912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:43.407897 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:43.408690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:43.408671 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:18:43.593246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:43.593205 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:44.326962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:44.326915 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:45.205962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:45.205923 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:46.960796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:46.093250 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:46.960796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:46.758007 6208 generic.go:356] "Generic (PLEG): container finished" podID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerID="48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689" exitCode=0 Nov 04 00:18:46.960796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:46.758077 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" event={"ID":"1156db56-b84d-4241-9689-69a6aa1eaf5e","Type":"ContainerDied","Data":"48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689"} Nov 04 00:18:46.960796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:46.758841 6208 scope.go:122] "RemoveContainer" containerID="48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689" Nov 04 00:18:46.960796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:46.766795 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:46.960796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:46.766894 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:46.960796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:46.766940 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:46.960796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:46.797454 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:46.960796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:46.943632 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:47.771178 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.771140 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" event={"ID":"1156db56-b84d-4241-9689-69a6aa1eaf5e","Type":"ContainerStarted","Data":"cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60"} Nov 04 00:18:47.823597 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.823565 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z"] Nov 04 00:18:47.835389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.831129 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.875500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.875446 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z"] Nov 04 00:18:47.909196 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.908880 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.909196 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.909003 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.909196 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.909158 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515f512b-c7a2-4d2c-b443-bfe912c77f80-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.909476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.909267 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.909476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.909302 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.909476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.909348 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.909476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.909378 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.909476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.909405 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29b4f\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-kube-api-access-29b4f\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.909476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.909445 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:47.951096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:47.950879 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:48.019419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.018942 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:18:48.070143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.068916 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29b4f\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-kube-api-access-29b4f\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.070143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.069026 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.070143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.069432 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.070143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.069529 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.070143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.069602 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515f512b-c7a2-4d2c-b443-bfe912c77f80-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.070143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.069672 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.070143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.069717 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.070143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.069774 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.070143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.069803 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.073175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.071567 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 00:18:48.073175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.071636 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 00:18:48.073175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.072055 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.073370 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.073315 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.086057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.074001 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515f512b-c7a2-4d2c-b443-bfe912c77f80-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.086057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.076266 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.086057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.076400 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.086057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.076588 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.099375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.091607 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.099375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.096788 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.099375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.097187 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29b4f\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-kube-api-access-29b4f\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.287393 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.287347 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:48.684399 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.684369 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z"] Nov 04 00:18:48.694209 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:18:48.694174 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515f512b_c7a2_4d2c_b443_bfe912c77f80.slice/crio-113d52bfe41f09f11c74a34d55763b686b0f0536720599c0342cde0200bfd9af WatchSource:0}: Error finding container 113d52bfe41f09f11c74a34d55763b686b0f0536720599c0342cde0200bfd9af: Status 404 returned error can't find the container with id 113d52bfe41f09f11c74a34d55763b686b0f0536720599c0342cde0200bfd9af Nov 04 00:18:48.698761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.698331 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:48.698761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.698421 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:48.698761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.698466 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:48.789939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.789906 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" event={"ID":"515f512b-c7a2-4d2c-b443-bfe912c77f80","Type":"ContainerStarted","Data":"113d52bfe41f09f11c74a34d55763b686b0f0536720599c0342cde0200bfd9af"} Nov 04 00:18:48.940118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:48.940080 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:49.018978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.018533 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 00:18:49.018978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.018590 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 00:18:49.800539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.800496 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" event={"ID":"515f512b-c7a2-4d2c-b443-bfe912c77f80","Type":"ContainerStarted","Data":"afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a"} Nov 04 00:18:49.802513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.802484 6208 generic.go:356] "Generic (PLEG): container finished" podID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerID="3eb4f86e887ae23511d74fd2b3666eddc86fcf3c7f45618ee9de9a605f878654" exitCode=0 Nov 04 00:18:49.802625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.802553 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" event={"ID":"3fb44f32-c801-4a3b-b97b-81e776ddbbd6","Type":"ContainerDied","Data":"3eb4f86e887ae23511d74fd2b3666eddc86fcf3c7f45618ee9de9a605f878654"} Nov 04 00:18:49.803496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.803026 6208 scope.go:122] "RemoveContainer" containerID="3eb4f86e887ae23511d74fd2b3666eddc86fcf3c7f45618ee9de9a605f878654" Nov 04 00:18:49.805813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.805781 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:49.805914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.805854 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:49.805914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.805884 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:18:49.828307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.828258 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" podStartSLOduration=2.828243607 podStartE2EDuration="2.828243607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:18:49.825423915 +0000 UTC m=+2948.648020078" watchObservedRunningTime="2026-11-04 00:18:49.828243607 +0000 UTC m=+2948.650839769" Nov 04 00:18:49.993323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:49.993279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:50.018498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:50.018026 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 00:18:50.018498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:50.018113 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 00:18:50.289429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:50.289389 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:50.289860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:50.289830 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 00:18:50.290060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:50.290013 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 00:18:50.408078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:50.400151 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:18:50.818005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:50.817957 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" event={"ID":"3fb44f32-c801-4a3b-b97b-81e776ddbbd6","Type":"ContainerStarted","Data":"c5465e3d6ea30942d305d7f2260dac0b8fa21d82f1f06676fb049102ca0a6100"} Nov 04 00:18:51.018128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.018082 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 00:18:51.018364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.018147 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 00:18:51.044144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.044099 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:18:51.080122 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.074623 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:51.271062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.270406 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:18:51.283084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.276160 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 04 00:18:51.283084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.276228 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 04 00:18:51.355118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.354157 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 00:18:51.355118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.354225 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 00:18:51.942057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:51.941698 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:52.022058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:52.021219 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 00:18:52.022058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:52.021289 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 00:18:52.270058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:52.268399 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 04 00:18:52.270058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:52.268461 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 04 00:18:52.288617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:52.288574 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 00:18:52.288811 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:52.288650 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 00:18:52.854124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:52.854068 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:53.033062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:53.023823 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 00:18:53.033062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:53.023887 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 00:18:53.275058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:53.270485 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 04 00:18:53.275058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:53.270581 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 04 00:18:53.308059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:53.307856 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 00:18:53.308059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:53.307919 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 00:18:53.497061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:53.497004 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:53.947061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:53.943501 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:54.073924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:54.073853 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 00:18:54.081484 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:54.081440 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 00:18:54.294430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:54.294389 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 04 00:18:54.294715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:54.294685 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 04 00:18:54.295245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:54.295217 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 00:18:54.295441 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:54.295417 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 00:18:54.841676 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:54.841623 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:55.223214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:55.223111 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:18:55.230012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:55.229979 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:18:55.285058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:55.281324 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 04 00:18:55.285058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:55.281557 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 04 00:18:55.329159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:55.326443 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 04 00:18:55.329159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:55.326503 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 04 00:18:55.619806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:55.619706 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:56.281058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:56.274453 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 04 00:18:56.281058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:56.274535 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 04 00:18:56.436656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:56.436568 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:56.437949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:56.437439 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:56.438611 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:56.438426 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:56.441123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:56.441066 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:18:57.184476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:57.184441 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:57.334303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:57.334272 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:18:57.335171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:57.335147 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:18:57.336491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:57.336469 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:18:58.184058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:58.179694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:18:59.288061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:18:59.287280 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:00.158631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:00.158569 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:01.126018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:01.125984 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:01.829824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:01.829788 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:02.540954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:02.540911 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:03.585205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:03.585103 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:03.696321 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:03.696265 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:04.222290 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.222249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:04.459908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.459866 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr"] Nov 04 00:19:04.473071 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.472968 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.476633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.476534 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr"] Nov 04 00:19:04.487659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.487625 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8895cb\"" Nov 04 00:19:04.490711 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.489380 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"routing-overlay\"" Nov 04 00:19:04.490955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.489437 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"agc-cred-86bb370d5296\"" Nov 04 00:19:04.491090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.489527 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"openshift-service-ca.crt\"" Nov 04 00:19:04.491191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.489592 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"payload-processing-external-model-e2e--9366bb2e-dockercfg-z9lrg\"" Nov 04 00:19:04.579634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.579335 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8895cb\"/\"kube-root-ca.crt\"" Nov 04 00:19:04.597062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.597006 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4htxr\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-kube-api-access-4htxr\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.597479 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.597305 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.597479 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.597366 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.598204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.597481 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.699347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.699310 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.699534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.699390 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4htxr\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-kube-api-access-4htxr\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.699580 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.699569 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.699622 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.699612 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.700284 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.700252 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.700422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.700263 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.702815 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.702789 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.712385 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.712352 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4htxr\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-kube-api-access-4htxr\") pod \"payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.724721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.724632 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx"] Nov 04 00:19:04.729919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.729893 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.735419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.735343 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"agc-cred-c54e0cfe3210\"" Nov 04 00:19:04.735585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.735459 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"kube-root-ca.crt\"" Nov 04 00:19:04.735585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.735565 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"payload-processing-external-model-e2e--f0c21228-dockercfg-c5cmt\"" Nov 04 00:19:04.736186 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.736167 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"routing-overlay\"" Nov 04 00:19:04.736267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.736214 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-06ade4\"" Nov 04 00:19:04.736316 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.736286 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-06ade4\"/\"openshift-service-ca.crt\"" Nov 04 00:19:04.740430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.740397 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx"] Nov 04 00:19:04.801809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.801590 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.802812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.801950 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtk7z\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-kube-api-access-dtk7z\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.802812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.802086 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.802812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.802208 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.810800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.810773 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:04.903986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.903942 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtk7z\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-kube-api-access-dtk7z\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.904156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.904109 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.904690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.904260 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.904690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.904376 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.904927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.904849 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.905891 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.905803 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.918159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.918128 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:04.922276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:04.922226 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtk7z\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-kube-api-access-dtk7z\") pod \"payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:05.044875 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:05.044778 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:05.067568 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:05.066374 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:05.222401 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:19:05.221899 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b43ebf7_9e03_4596_8143_987063fcb666.slice/crio-7afaf7a9a9c1f204cfe434cac33439f06a3bc9a6acce9f2bc83a1440eb90ce0a WatchSource:0}: Error finding container 7afaf7a9a9c1f204cfe434cac33439f06a3bc9a6acce9f2bc83a1440eb90ce0a: Status 404 returned error can't find the container with id 7afaf7a9a9c1f204cfe434cac33439f06a3bc9a6acce9f2bc83a1440eb90ce0a Nov 04 00:19:05.222401 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:05.222333 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr"] Nov 04 00:19:05.422631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:05.422591 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" event={"ID":"3b43ebf7-9e03-4596-8143-987063fcb666","Type":"ContainerStarted","Data":"130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f"} Nov 04 00:19:05.422835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:05.422649 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" event={"ID":"3b43ebf7-9e03-4596-8143-987063fcb666","Type":"ContainerStarted","Data":"7afaf7a9a9c1f204cfe434cac33439f06a3bc9a6acce9f2bc83a1440eb90ce0a"} Nov 04 00:19:05.422835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:05.422729 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:05.445653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:05.445600 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" podStartSLOduration=1.445584604 podStartE2EDuration="1.445584604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:05.443948645 +0000 UTC m=+2964.266544809" watchObservedRunningTime="2026-11-04 00:19:05.445584604 +0000 UTC m=+2964.268180782" Nov 04 00:19:05.520718 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:05.520686 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx"] Nov 04 00:19:05.521789 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:19:05.521138 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f37911_cb19_40b9_9728_c9331d9a3937.slice/crio-196ad088b5a7be84c8e580da5168a2ee1555d45711992de0e9abb4e348d59e58 WatchSource:0}: Error finding container 196ad088b5a7be84c8e580da5168a2ee1555d45711992de0e9abb4e348d59e58: Status 404 returned error can't find the container with id 196ad088b5a7be84c8e580da5168a2ee1555d45711992de0e9abb4e348d59e58 Nov 04 00:19:05.708942 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:19:05.708893 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:19:05.989821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:05.989784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:06.432361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:06.432273 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" event={"ID":"f7f37911-cb19-40b9-9728-c9331d9a3937","Type":"ContainerStarted","Data":"e0c2facb2a77f16b49c883f7ea0e6df005d863b86daee864fd74ba362fe90959"} Nov 04 00:19:06.432361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:06.432331 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" event={"ID":"f7f37911-cb19-40b9-9728-c9331d9a3937","Type":"ContainerStarted","Data":"196ad088b5a7be84c8e580da5168a2ee1555d45711992de0e9abb4e348d59e58"} Nov 04 00:19:06.433904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:06.432517 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:06.481418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:06.481356 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" podStartSLOduration=2.481340147 podStartE2EDuration="2.481340147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:06.450557728 +0000 UTC m=+2965.273153890" watchObservedRunningTime="2026-11-04 00:19:06.481340147 +0000 UTC m=+2965.303936309" Nov 04 00:19:06.728347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:06.728309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:07.522550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:07.522509 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:07.880365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:07.880265 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:08.332144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:08.332102 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:08.456284 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:08.456244 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:08.778267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:08.778225 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j"] Nov 04 00:19:08.784175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:08.784147 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:08.792061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:08.791409 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-757a7d-openshift-default-dockercfg-4fwxb\"" Nov 04 00:19:08.810627 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:08.810583 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:08.811678 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:08.811657 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j"] Nov 04 00:19:09.061118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.048047 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.061118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.048133 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-credential-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.061118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.048246 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-envoy\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.061118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.048323 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-podinfo\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.061118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.048395 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85cdp\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-kube-api-access-85cdp\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.061118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.048505 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-data\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.061118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.048593 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-token\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.061118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.048712 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-certs\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.061118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.048765 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istiod-ca-cert\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.166696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.166430 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-certs\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.166696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.166508 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istiod-ca-cert\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.166696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.166602 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.166696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.166646 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-credential-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.167108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.166735 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-envoy\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.167108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.166800 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-podinfo\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.167108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.166863 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85cdp\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-kube-api-access-85cdp\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.167108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.166956 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-data\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.167108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.166987 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-token\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.197066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.167827 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-certs\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.197896 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.197743 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.198157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.198129 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istiod-ca-cert\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.198487 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.198444 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-data\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.198830 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.198807 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-credential-socket\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.235894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.235807 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85cdp\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-kube-api-access-85cdp\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.255070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.255021 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-envoy\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.268918 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.262446 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-podinfo\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.269182 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.267827 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-token\") pod \"e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.285671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.285634 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:09.410940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.410836 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:09.438595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.438559 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:09.831574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.831537 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j"] Nov 04 00:19:09.832104 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:19:09.832076 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a4cc55_4497_4f36_aecd_cd7e5e21a0bd.slice/crio-413c1446e8902c98a80a53f5cfee0e722318013fcf4931a2175d3ab886e67cd8 WatchSource:0}: Error finding container 413c1446e8902c98a80a53f5cfee0e722318013fcf4931a2175d3ab886e67cd8: Status 404 returned error can't find the container with id 413c1446e8902c98a80a53f5cfee0e722318013fcf4931a2175d3ab886e67cd8 Nov 04 00:19:09.835471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.835434 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:09.835580 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.835507 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:09.835580 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:09.835537 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:10.433542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:10.433494 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:10.558064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:10.554743 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" event={"ID":"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd","Type":"ContainerStarted","Data":"d9a516e3a4149a8d3202431d6c6fa72efbb5190e9065f212c72273b4270ec4e2"} Nov 04 00:19:10.558064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:10.554818 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" event={"ID":"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd","Type":"ContainerStarted","Data":"413c1446e8902c98a80a53f5cfee0e722318013fcf4931a2175d3ab886e67cd8"} Nov 04 00:19:10.593706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:10.593633 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" podStartSLOduration=2.593616184 podStartE2EDuration="2.593616184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:10.588974604 +0000 UTC m=+2969.411570777" watchObservedRunningTime="2026-11-04 00:19:10.593616184 +0000 UTC m=+2969.416212346" Nov 04 00:19:11.314106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:11.314060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:11.439897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:11.437104 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:11.478099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:11.477367 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:11.565259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:11.565172 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:11.646255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:11.566256 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:19:12.178809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:12.178769 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:12.578221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:12.578170 6208 generic.go:356] "Generic (PLEG): container finished" podID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerID="72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2" exitCode=0 Nov 04 00:19:12.578660 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:12.578309 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" event={"ID":"1b1903fe-748c-4b83-9d58-7727b1f9b755","Type":"ContainerDied","Data":"72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2"} Nov 04 00:19:12.578995 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:12.578978 6208 scope.go:122] "RemoveContainer" containerID="72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2" Nov 04 00:19:12.581045 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:12.581009 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:12.581127 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:12.581095 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:12.582817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:12.581129 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:12.906565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:12.906525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:13.006204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:13.006161 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:13.608166 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:13.608120 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" event={"ID":"1b1903fe-748c-4b83-9d58-7727b1f9b755","Type":"ContainerStarted","Data":"50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93"} Nov 04 00:19:13.950124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:13.949984 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:14.234115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:14.234073 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:14.334530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:14.334497 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:14.447056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:14.445523 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:19:14.448570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:14.448538 6208 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 00:19:14.448797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:14.448770 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 00:19:15.236368 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:15.236329 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:15.418492 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:15.418442 6208 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 00:19:15.418679 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:15.418519 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 00:19:16.319792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:16.319743 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:16.418983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:16.418474 6208 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 00:19:16.418983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:16.418561 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 00:19:16.480723 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:16.479270 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:17.079707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.079667 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:17.416289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.416190 6208 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 00:19:17.416289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.416268 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 00:19:17.444289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.444261 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:17.890237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.890197 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh"] Nov 04 00:19:17.896653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.896622 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.902495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.902455 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh"] Nov 04 00:19:17.928316 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.928283 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-kube-api-access-vdlrs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.928611 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.928565 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.928734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.928699 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390a996-22ca-4972-90d7-c2c6f104c4cc-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.928861 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.928842 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.928991 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.928974 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.929188 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.929162 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.929309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.929211 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.929309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.929303 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.929429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.929356 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:17.938204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:17.938177 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.082333 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-kube-api-access-vdlrs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.082411 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.082474 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390a996-22ca-4972-90d7-c2c6f104c4cc-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.082626 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.082708 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.082842 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.082879 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.082941 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.083006 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.099615 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-data\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.102760 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390a996-22ca-4972-90d7-c2c6f104c4cc-istiod-ca-cert\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.103069 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-credential-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.106057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.103603 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-socket\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.118461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.118245 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-token\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.119217 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.100907 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-certs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.119451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.119424 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-podinfo\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.120245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.120105 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-kube-api-access-vdlrs\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.120332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.120299 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-envoy\") pod \"e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.224671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.224635 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:18.416192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.416153 6208 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 00:19:18.416368 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.416226 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 00:19:18.545565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.545535 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh"] Nov 04 00:19:18.546239 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:19:18.546212 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8390a996_22ca_4972_90d7_c2c6f104c4cc.slice/crio-4b526035bfcbe1ec537f76dd39b88e408c4d3ffe9a771c8c6a2215856e7b3d5e WatchSource:0}: Error finding container 4b526035bfcbe1ec537f76dd39b88e408c4d3ffe9a771c8c6a2215856e7b3d5e: Status 404 returned error can't find the container with id 4b526035bfcbe1ec537f76dd39b88e408c4d3ffe9a771c8c6a2215856e7b3d5e Nov 04 00:19:18.550107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.550013 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:18.550187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.550147 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:18.550187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.550183 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:18.808950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.808851 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:18.819068 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.819024 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" event={"ID":"8390a996-22ca-4972-90d7-c2c6f104c4cc","Type":"ContainerStarted","Data":"5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b"} Nov 04 00:19:18.819237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.819088 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" event={"ID":"8390a996-22ca-4972-90d7-c2c6f104c4cc","Type":"ContainerStarted","Data":"4b526035bfcbe1ec537f76dd39b88e408c4d3ffe9a771c8c6a2215856e7b3d5e"} Nov 04 00:19:18.846807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:18.846744 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" podStartSLOduration=1.84672538 podStartE2EDuration="1.84672538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:18.842660158 +0000 UTC m=+2977.665256335" watchObservedRunningTime="2026-11-04 00:19:18.84672538 +0000 UTC m=+2977.669321544" Nov 04 00:19:19.191075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:19.190954 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:19:19.225825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:19.225787 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:19.227529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:19.227500 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 00:19:19.227659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:19.227555 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 00:19:19.416054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:19.415996 6208 patch_prober.go:29] interesting pod/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 04 00:19:19.416248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:19.416068 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 04 00:19:19.733272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:19.733176 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:20.225027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:20.224994 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 00:19:20.225247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:20.225088 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 00:19:20.421245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:20.421215 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:19:20.422468 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:20.422450 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:19:20.646566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:20.646466 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:21.225114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:21.225073 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 00:19:21.225529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:21.225127 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 00:19:21.537539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:21.537451 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:22.225466 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:22.225424 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 00:19:22.227006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:22.225494 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 00:19:22.378667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:22.378629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:23.225879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:23.225844 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 00:19:23.226307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:23.225906 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 00:19:23.280862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:23.280826 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:24.035991 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:24.035951 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:24.225728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:24.225693 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 00:19:24.225893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:24.225751 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 00:19:24.932686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:24.932629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:25.225073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:25.225019 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 04 00:19:25.225292 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:25.225112 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 04 00:19:26.007060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.003213 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:26.231736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.231705 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:26.232279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.232253 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:26.233091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.233073 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:19:26.273549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.273466 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr"] Nov 04 00:19:26.312545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.312512 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:26.409419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.409373 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:26.792384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.789816 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx"] Nov 04 00:19:26.803528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.803436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:26.915846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:26.915811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:27.031938 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:27.031903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:27.526921 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:27.526446 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:28.049752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:28.049702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:28.760622 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:28.760591 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:29.462555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:29.462520 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:30.247461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:30.247426 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:30.968066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:30.968016 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:31.274056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.273942 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" podUID="3b43ebf7-9e03-4596-8143-987063fcb666" containerName="payload-processing" containerID="cri-o://130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f" gracePeriod=25 Nov 04 00:19:31.569624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.569598 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:31.628416 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.628380 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:31.660130 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.660096 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-plugins-config-volume\") pod \"3b43ebf7-9e03-4596-8143-987063fcb666\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " Nov 04 00:19:31.660304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.660212 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-provider-credentials\") pod \"3b43ebf7-9e03-4596-8143-987063fcb666\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " Nov 04 00:19:31.660384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.660355 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-routing-overlay-volume\") pod \"3b43ebf7-9e03-4596-8143-987063fcb666\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " Nov 04 00:19:31.660607 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.660571 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-plugins-config-volume" pod "3b43ebf7-9e03-4596-8143-987063fcb666" (UID: "3b43ebf7-9e03-4596-8143-987063fcb666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:19:31.660724 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.660690 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-kube-api-access-4htxr\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-kube-api-access-4htxr\") pod \"3b43ebf7-9e03-4596-8143-987063fcb666\" (UID: \"3b43ebf7-9e03-4596-8143-987063fcb666\") " Nov 04 00:19:31.660724 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.660708 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-routing-overlay-volume" pod "3b43ebf7-9e03-4596-8143-987063fcb666" (UID: "3b43ebf7-9e03-4596-8143-987063fcb666"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:19:31.661354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.661333 6208 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-routing-overlay-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:19:31.661354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.661355 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b43ebf7-9e03-4596-8143-987063fcb666-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:19:31.663958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.663937 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-kube-api-access-4htxr" pod "3b43ebf7-9e03-4596-8143-987063fcb666" (UID: "3b43ebf7-9e03-4596-8143-987063fcb666"). InnerVolumeSpecName "kube-api-access-4htxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:19:31.664076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.663971 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-provider-credentials" pod "3b43ebf7-9e03-4596-8143-987063fcb666" (UID: "3b43ebf7-9e03-4596-8143-987063fcb666"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:19:31.763616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.763581 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4htxr\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-kube-api-access-4htxr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:19:31.763616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.763610 6208 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b43ebf7-9e03-4596-8143-987063fcb666-provider-credentials\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:19:31.795083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.794987 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" podUID="f7f37911-cb19-40b9-9728-c9331d9a3937" containerName="payload-processing" containerID="cri-o://e0c2facb2a77f16b49c883f7ea0e6df005d863b86daee864fd74ba362fe90959" gracePeriod=25 Nov 04 00:19:31.836017 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:31.835982 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:32.128731 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.128645 6208 generic.go:356] "Generic (PLEG): container finished" podID="3b43ebf7-9e03-4596-8143-987063fcb666" containerID="130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f" exitCode=0 Nov 04 00:19:32.128731 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.128695 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" Nov 04 00:19:32.129304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.128725 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" event={"ID":"3b43ebf7-9e03-4596-8143-987063fcb666","Type":"ContainerDied","Data":"130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f"} Nov 04 00:19:32.129304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.128789 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr" event={"ID":"3b43ebf7-9e03-4596-8143-987063fcb666","Type":"ContainerDied","Data":"7afaf7a9a9c1f204cfe434cac33439f06a3bc9a6acce9f2bc83a1440eb90ce0a"} Nov 04 00:19:32.129304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.128819 6208 scope.go:122] "RemoveContainer" containerID="130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f" Nov 04 00:19:32.132133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.132106 6208 generic.go:356] "Generic (PLEG): container finished" podID="f7f37911-cb19-40b9-9728-c9331d9a3937" containerID="e0c2facb2a77f16b49c883f7ea0e6df005d863b86daee864fd74ba362fe90959" exitCode=0 Nov 04 00:19:32.132247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.132184 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" event={"ID":"f7f37911-cb19-40b9-9728-c9331d9a3937","Type":"ContainerDied","Data":"e0c2facb2a77f16b49c883f7ea0e6df005d863b86daee864fd74ba362fe90959"} Nov 04 00:19:32.148447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.148424 6208 scope.go:122] "RemoveContainer" containerID="130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f" Nov 04 00:19:32.148735 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:19:32.148710 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f\": container with ID starting with 130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f not found: ID does not exist" containerID="130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f" Nov 04 00:19:32.149744 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.148743 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f"} err="failed to get container status \"130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f\": rpc error: code = NotFound desc = could not find container \"130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f\": container with ID starting with 130fac4dec67476cc8e2d26f878581ac9d830e32d0e239b0744667e529239c1f not found: ID does not exist" Nov 04 00:19:32.152361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.152336 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:32.166292 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.166266 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr"] Nov 04 00:19:32.170699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.170678 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8895cb/payload-processing-external-model-e2e-extmod-a-8895cb-74577wwsr"] Nov 04 00:19:32.285476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.285401 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-plugins-config-volume\") pod \"f7f37911-cb19-40b9-9728-c9331d9a3937\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " Nov 04 00:19:32.286054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.285894 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-kube-api-access-dtk7z\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-kube-api-access-dtk7z\") pod \"f7f37911-cb19-40b9-9728-c9331d9a3937\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " Nov 04 00:19:32.287279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.286269 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-routing-overlay-volume\") pod \"f7f37911-cb19-40b9-9728-c9331d9a3937\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " Nov 04 00:19:32.287279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.286326 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-provider-credentials\") pod \"f7f37911-cb19-40b9-9728-c9331d9a3937\" (UID: \"f7f37911-cb19-40b9-9728-c9331d9a3937\") " Nov 04 00:19:32.291293 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.291236 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-provider-credentials" pod "f7f37911-cb19-40b9-9728-c9331d9a3937" (UID: "f7f37911-cb19-40b9-9728-c9331d9a3937"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:19:32.292087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.291802 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-plugins-config-volume" pod "f7f37911-cb19-40b9-9728-c9331d9a3937" (UID: "f7f37911-cb19-40b9-9728-c9331d9a3937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:19:32.297674 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.297646 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-routing-overlay-volume" pod "f7f37911-cb19-40b9-9728-c9331d9a3937" (UID: "f7f37911-cb19-40b9-9728-c9331d9a3937"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:19:32.299460 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.299434 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-kube-api-access-dtk7z" pod "f7f37911-cb19-40b9-9728-c9331d9a3937" (UID: "f7f37911-cb19-40b9-9728-c9331d9a3937"). InnerVolumeSpecName "kube-api-access-dtk7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:19:32.390443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.390366 6208 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-routing-overlay-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:19:32.390443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.390399 6208 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-provider-credentials\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:19:32.390443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.390414 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f37911-cb19-40b9-9728-c9331d9a3937-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:19:32.390443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.390429 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtk7z\" (UniqueName: \"kubernetes.io/projected/f7f37911-cb19-40b9-9728-c9331d9a3937-kube-api-access-dtk7z\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:19:32.657323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:32.657235 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:33.142364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:33.142327 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" event={"ID":"f7f37911-cb19-40b9-9728-c9331d9a3937","Type":"ContainerDied","Data":"196ad088b5a7be84c8e580da5168a2ee1555d45711992de0e9abb4e348d59e58"} Nov 04 00:19:33.142364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:33.142349 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx" Nov 04 00:19:33.143533 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:33.142390 6208 scope.go:122] "RemoveContainer" containerID="e0c2facb2a77f16b49c883f7ea0e6df005d863b86daee864fd74ba362fe90959" Nov 04 00:19:33.169104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:33.169065 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx"] Nov 04 00:19:33.174527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:33.174495 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-06ade4/payload-processing-external-model-e2e-extmod-b-06ade4-6bf4z8ppx"] Nov 04 00:19:33.297583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:33.297539 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b43ebf7-9e03-4596-8143-987063fcb666" path="/var/lib/kubelet/pods/3b43ebf7-9e03-4596-8143-987063fcb666/volumes" Nov 04 00:19:33.298190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:33.298167 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f37911-cb19-40b9-9728-c9331d9a3937" path="/var/lib/kubelet/pods/f7f37911-cb19-40b9-9728-c9331d9a3937/volumes" Nov 04 00:19:33.362604 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:33.362566 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:34.109486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:34.109449 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:34.219170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:34.219131 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:34.342058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:34.341654 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:34.635788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:34.635725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:35.049594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.049555 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw"] Nov 04 00:19:35.050563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.050535 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3b43ebf7-9e03-4596-8143-987063fcb666" containerName="payload-processing" Nov 04 00:19:35.050563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.050562 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b43ebf7-9e03-4596-8143-987063fcb666" containerName="payload-processing" Nov 04 00:19:35.050753 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.050600 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f7f37911-cb19-40b9-9728-c9331d9a3937" containerName="payload-processing" Nov 04 00:19:35.050753 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.050609 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f37911-cb19-40b9-9728-c9331d9a3937" containerName="payload-processing" Nov 04 00:19:35.050859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.050833 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b43ebf7-9e03-4596-8143-987063fcb666" containerName="payload-processing" Nov 04 00:19:35.050859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.050853 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f37911-cb19-40b9-9728-c9331d9a3937" containerName="payload-processing" Nov 04 00:19:35.056335 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.056312 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.067374 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.067318 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw"] Nov 04 00:19:35.089970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.089943 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istiod-ca-cert\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.090763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.090065 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-podinfo\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.090763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.090119 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-credential-socket\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.090763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.090272 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2nt\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-kube-api-access-7t2nt\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.090763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.090328 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-token\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.090763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.090712 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-envoy\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.091123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.090808 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-data\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.091123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.090922 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-socket\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.091387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.091140 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-certs\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.092277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.092248 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:35.112377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.112356 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ecd46b-openshift-default-dockercfg-cfq2g\"" Nov 04 00:19:35.192024 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.191994 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:35.192316 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192294 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-credential-socket\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.192380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192366 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2nt\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-kube-api-access-7t2nt\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.192444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192430 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-token\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.192485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192477 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-envoy\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.192566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192551 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-data\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.192683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192667 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-socket\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.192748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192729 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-credential-socket\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.192848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192830 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-certs\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.192910 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192866 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istiod-ca-cert\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.192983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192955 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-podinfo\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.193079 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.192994 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-data\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.193079 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.193028 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-socket\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.193221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.193201 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-certs\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.193634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.193608 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istiod-ca-cert\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.195149 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.195128 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-envoy\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.195397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.195380 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-podinfo\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.204571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.204543 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-token\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.204677 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.204574 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2nt\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-kube-api-access-7t2nt\") pod \"e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.380495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.380405 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:35.721651 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:19:35.721584 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:19:35.929909 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:19:35.929875 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2762e9_da42_47f7_b645_6dc0233f3f8d.slice/crio-e52d0a0266daaee6aaf16a6f095f11e407ade498fabaa06cd911300b99019e9b WatchSource:0}: Error finding container e52d0a0266daaee6aaf16a6f095f11e407ade498fabaa06cd911300b99019e9b: Status 404 returned error can't find the container with id e52d0a0266daaee6aaf16a6f095f11e407ade498fabaa06cd911300b99019e9b Nov 04 00:19:35.930332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.930309 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw"] Nov 04 00:19:35.933780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.933742 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:35.933873 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.933831 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:35.933928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:35.933885 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:19:36.036747 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:36.036715 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:36.181777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:36.181456 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" event={"ID":"8a2762e9-da42-47f7-b645-6dc0233f3f8d","Type":"ContainerStarted","Data":"cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc"} Nov 04 00:19:36.181777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:36.181515 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" event={"ID":"8a2762e9-da42-47f7-b645-6dc0233f3f8d","Type":"ContainerStarted","Data":"e52d0a0266daaee6aaf16a6f095f11e407ade498fabaa06cd911300b99019e9b"} Nov 04 00:19:36.207108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:36.207047 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" podStartSLOduration=1.207010022 podStartE2EDuration="1.207010022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:36.204164335 +0000 UTC m=+2995.026760500" watchObservedRunningTime="2026-11-04 00:19:36.207010022 +0000 UTC m=+2995.029606185" Nov 04 00:19:36.401159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:36.401119 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:36.904833 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:36.904793 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:37.393294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:37.393264 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:37.730171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:37.730132 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:38.198691 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:38.198609 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:38.199697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:38.199673 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:19:38.493300 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:38.493267 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:38.597218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:38.597172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:39.628960 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:39.628924 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:39.732137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:39.732096 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:39.830617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:39.830585 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:40.640796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:40.640763 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:41.382881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.377313 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:41.396123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.396089 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt"] Nov 04 00:19:41.439213 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.439185 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt"] Nov 04 00:19:41.439491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.439475 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.447180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.447150 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-757a7d-serving-cert\"" Nov 04 00:19:41.488512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.488481 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:41.629478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.629438 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:41.629769 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.629485 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-maas-api-tls\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.630174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.630151 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-metrics-certs\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.630308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.630223 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq28\" (UniqueName: \"kubernetes.io/projected/a6081121-890a-43f4-b85f-2bc166d06035-kube-api-access-vvq28\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.630425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.630407 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6081121-890a-43f4-b85f-2bc166d06035-openshift-service-ca\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.732314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.732276 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6081121-890a-43f4-b85f-2bc166d06035-openshift-service-ca\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.732776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.732460 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-maas-api-tls\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.732776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.732602 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-metrics-certs\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.732776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.732643 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq28\" (UniqueName: \"kubernetes.io/projected/a6081121-890a-43f4-b85f-2bc166d06035-kube-api-access-vvq28\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.732958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.732875 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6081121-890a-43f4-b85f-2bc166d06035-openshift-service-ca\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.735184 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.735138 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-maas-api-tls\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.735409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.735392 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-metrics-certs\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.745077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.745052 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq28\" (UniqueName: \"kubernetes.io/projected/a6081121-890a-43f4-b85f-2bc166d06035-kube-api-access-vvq28\") pod \"maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:41.761917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:41.761897 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:42.130324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:42.125707 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt"] Nov 04 00:19:42.130324 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:19:42.126230 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6081121_890a_43f4_b85f_2bc166d06035.slice/crio-856a799836e726ea46ebabf07d69c096f53fb2e9b48479f3a0afc7cf844e2c80 WatchSource:0}: Error finding container 856a799836e726ea46ebabf07d69c096f53fb2e9b48479f3a0afc7cf844e2c80: Status 404 returned error can't find the container with id 856a799836e726ea46ebabf07d69c096f53fb2e9b48479f3a0afc7cf844e2c80 Nov 04 00:19:42.268255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:42.268215 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" event={"ID":"a6081121-890a-43f4-b85f-2bc166d06035","Type":"ContainerStarted","Data":"856a799836e726ea46ebabf07d69c096f53fb2e9b48479f3a0afc7cf844e2c80"} Nov 04 00:19:42.365570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:42.365523 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:42.828618 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:42.828595 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:19:42.828950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:42.828595 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:19:43.152561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:43.152468 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:43.304416 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:43.304385 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:43.304594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:43.304434 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" event={"ID":"a6081121-890a-43f4-b85f-2bc166d06035","Type":"ContainerStarted","Data":"61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2"} Nov 04 00:19:43.322484 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:43.322432 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" podStartSLOduration=1.634147537 podStartE2EDuration="2.322418721s" totalImagesPullingTime="688.271184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:19:43.320442339 +0000 UTC m=+3002.143038501" watchObservedRunningTime="2026-11-04 00:19:43.322418721 +0000 UTC m=+3002.145014883" Nov 04 00:19:43.913707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:43.913667 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:44.687819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:44.687778 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:45.437877 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:45.437838 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:46.306059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:46.302973 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:47.274629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:47.274589 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:48.129445 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:48.129395 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:48.939269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:48.939233 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:49.315430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:49.315403 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:19:49.744445 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:49.744412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:50.540651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:50.540611 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:51.244798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:51.244766 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:52.042577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:52.042543 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:52.837105 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:52.837075 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:53.527510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:53.527472 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:53.634649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:53.634610 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:54.421204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:54.421166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:55.123471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:55.123437 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:55.852372 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:55.852336 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:56.646322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:56.646284 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:57.496655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:57.496610 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:58.225389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:58.225338 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:58.943329 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:58.943285 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:19:59.763210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:19:59.763143 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:00.628884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:00.628853 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:01.455673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:01.455641 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:01.561418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:01.561374 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:02.201917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:02.201877 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:02.970227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:02.970193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:03.920843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:03.920806 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:04.728669 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:04.728631 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:05.544435 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:05.544392 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:05.643506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:05.643457 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:05.833126 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:20:05.833002 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:20:06.428874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:06.428835 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:07.237479 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:07.237445 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:08.052647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:08.052612 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:08.918448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:08.918415 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:09.636418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:09.636370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:10.630092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:10.630024 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:11.430538 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:11.430499 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:12.128195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:12.128154 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:12.855831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:12.855790 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:13.644502 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:13.644470 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:13.992930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:13.992895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:14.371510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:14.371416 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:15.231430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:15.231253 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:15.332512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:15.332475 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:16.129798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:16.129754 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:16.895308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:16.895270 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:17.807630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:17.807583 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:18.516990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:18.516942 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:19.319736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:19.319705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:20.124666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:20.124629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:20.936818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:20.936782 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:21.635086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:21.635047 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:22.520408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:22.520376 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:23.423047 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:23.422997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:24.141511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:24.141476 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:24.950844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:24.950798 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:25.843997 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:25.843964 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:26.630629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:26.630597 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:26.997229 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:26.997183 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:27.525766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:27.525726 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:27.921376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:27.921277 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:28.438926 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:28.438889 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:29.247778 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:29.247737 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:30.245486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:30.245450 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:31.110649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:31.110606 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:31.230149 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:31.221519 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:32.149947 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:32.149913 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:33.004388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:33.004357 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:33.775205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:33.775114 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:34.740355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:34.740315 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:35.585081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:35.584649 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:35.985120 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:20:35.967395 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:20:36.341298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:36.341216 6208 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal" podUID="92a1bdd4dac917849714e10bd3b332b6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Nov 04 00:20:36.350854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:36.350828 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:37.071494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:37.071457 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:37.942176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:37.942128 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:38.677933 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:38.677836 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:39.527290 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:39.527253 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:40.238381 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:40.238344 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:40.950593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:40.950548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:41.773983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:41.773946 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:42.596756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:42.596718 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:42.607217 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:42.607178 6208 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:20:42.607394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:42.607242 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:20:42.979097 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:42.979025 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:43.327069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:43.326979 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:44.157898 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:44.157849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:44.473647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:44.473609 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:44.976349 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:44.976312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:45.105115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:45.094293 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:45.603741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:45.603701 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:46.057472 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:46.057436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:46.157889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:46.157856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:46.942986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:46.942952 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:47.827013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:47.826974 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:48.770899 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:48.770846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:49.346117 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:49.346068 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:49.441760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:49.441719 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:49.703508 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:49.703384 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:50.566379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.566335 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:50.630386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.630335 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn"] Nov 04 00:20:50.647925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.647890 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:20:50.649155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.648493 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn"] Nov 04 00:20:50.656172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.655389 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6382fc-dockercfg-bbj4d\"" Nov 04 00:20:50.671135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.671098 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:50.740486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.706585 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6382fc\"" Nov 04 00:20:50.878155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.875504 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfjm\" (UniqueName: \"kubernetes.io/projected/3974cd05-5e4b-4bb8-973a-648db4082546-kube-api-access-jlfjm\") pod \"payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn\" (UID: \"3974cd05-5e4b-4bb8-973a-648db4082546\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:20:50.878155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.875739 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3974cd05-5e4b-4bb8-973a-648db4082546-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn\" (UID: \"3974cd05-5e4b-4bb8-973a-648db4082546\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:20:50.953791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.953616 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:50.986529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.986488 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfjm\" (UniqueName: \"kubernetes.io/projected/3974cd05-5e4b-4bb8-973a-648db4082546-kube-api-access-jlfjm\") pod \"payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn\" (UID: \"3974cd05-5e4b-4bb8-973a-648db4082546\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:20:50.987004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.986976 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3974cd05-5e4b-4bb8-973a-648db4082546-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn\" (UID: \"3974cd05-5e4b-4bb8-973a-648db4082546\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:20:50.997549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:50.997514 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3974cd05-5e4b-4bb8-973a-648db4082546-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn\" (UID: \"3974cd05-5e4b-4bb8-973a-648db4082546\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:20:51.103089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:51.103054 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfjm\" (UniqueName: \"kubernetes.io/projected/3974cd05-5e4b-4bb8-973a-648db4082546-kube-api-access-jlfjm\") pod \"payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn\" (UID: \"3974cd05-5e4b-4bb8-973a-648db4082546\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:20:51.280743 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:51.280703 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:20:51.749480 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:51.749444 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:52.554423 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:52.126080 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn"] Nov 04 00:20:52.554423 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:20:52.127663 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3974cd05_5e4b_4bb8_973a_648db4082546.slice/crio-7da6a45e6c56e674379c4ce1f46d316a204f633afefdd9a393ecbc42fb50a8f2 WatchSource:0}: Error finding container 7da6a45e6c56e674379c4ce1f46d316a204f633afefdd9a393ecbc42fb50a8f2: Status 404 returned error can't find the container with id 7da6a45e6c56e674379c4ce1f46d316a204f633afefdd9a393ecbc42fb50a8f2 Nov 04 00:20:52.554423 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:52.419322 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" event={"ID":"3974cd05-5e4b-4bb8-973a-648db4082546","Type":"ContainerStarted","Data":"7da6a45e6c56e674379c4ce1f46d316a204f633afefdd9a393ecbc42fb50a8f2"} Nov 04 00:20:52.687499 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:52.687462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:53.407808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.407774 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:53.430803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.429167 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" event={"ID":"3974cd05-5e4b-4bb8-973a-648db4082546","Type":"ContainerStarted","Data":"e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c"} Nov 04 00:20:53.430803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.429267 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:20:53.430956 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.430927 6208 generic.go:356] "Generic (PLEG): container finished" podID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerID="d9a516e3a4149a8d3202431d6c6fa72efbb5190e9065f212c72273b4270ec4e2" exitCode=0 Nov 04 00:20:53.430997 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.430977 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" event={"ID":"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd","Type":"ContainerDied","Data":"d9a516e3a4149a8d3202431d6c6fa72efbb5190e9065f212c72273b4270ec4e2"} Nov 04 00:20:53.431771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.431440 6208 scope.go:122] "RemoveContainer" containerID="d9a516e3a4149a8d3202431d6c6fa72efbb5190e9065f212c72273b4270ec4e2" Nov 04 00:20:53.433599 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.433571 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:20:53.433694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.433636 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:20:53.433694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.433672 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:20:53.533918 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.533841 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" podStartSLOduration=3.533816508 podStartE2EDuration="3.533816508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:20:53.447264997 +0000 UTC m=+3072.269861160" watchObservedRunningTime="2026-11-04 00:20:53.533816508 +0000 UTC m=+3072.356412673" Nov 04 00:20:53.686822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:53.654915 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:54.226140 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:54.226109 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:54.240186 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:54.240158 6208 patch_prober.go:29] interesting pod/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:20:54.245631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:54.240207 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:20:54.445700 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:54.445656 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" event={"ID":"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd","Type":"ContainerStarted","Data":"690cd326c8410dc4226d0ccef7aa0aa0fc7ac1f66f25507431ff26965c0821b2"} Nov 04 00:20:55.016913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:55.016584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:55.412065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:55.412001 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:20:55.478159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:55.413268 6208 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 00:20:55.478159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:55.413312 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 00:20:55.729543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:55.729510 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:56.411364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.411325 6208 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 00:20:56.411588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.411389 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 00:20:56.432712 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.432677 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:56.527401 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.527362 6208 generic.go:356] "Generic (PLEG): container finished" podID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerID="cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc" exitCode=0 Nov 04 00:20:56.528141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.528110 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" event={"ID":"8a2762e9-da42-47f7-b645-6dc0233f3f8d","Type":"ContainerDied","Data":"cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc"} Nov 04 00:20:56.528974 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.528899 6208 scope.go:122] "RemoveContainer" containerID="cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc" Nov 04 00:20:56.546881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.542465 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:20:56.548524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.548487 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:20:56.548756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.548732 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:20:56.594675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:56.594645 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:20:57.349453 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:57.349423 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:57.412382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:57.412343 6208 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 00:20:57.412685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:57.412651 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 00:20:57.596056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:57.594761 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" event={"ID":"8a2762e9-da42-47f7-b645-6dc0233f3f8d","Type":"ContainerStarted","Data":"16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084"} Nov 04 00:20:58.397188 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:58.397139 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:20:58.397438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:58.397302 6208 patch_prober.go:29] interesting pod/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 00:20:58.397438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:58.397350 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 00:20:58.424507 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:58.413242 6208 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 00:20:58.424507 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:58.413302 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 00:20:58.450058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:58.438158 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:59.222645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:59.222610 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:20:59.390052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:59.389425 6208 patch_prober.go:29] interesting pod/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 00:20:59.390052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:59.389487 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 00:20:59.420180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:59.420143 6208 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 00:20:59.422222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:20:59.420384 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 00:21:00.117051 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:00.116844 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:00.389113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:00.388404 6208 patch_prober.go:29] interesting pod/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 00:21:00.389113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:00.388473 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 00:21:00.411957 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:00.411914 6208 patch_prober.go:29] interesting pod/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Nov 04 00:21:00.412194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:00.411978 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Nov 04 00:21:00.939999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:00.939962 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:01.387279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:01.386641 6208 patch_prober.go:29] interesting pod/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 00:21:01.387279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:01.386700 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 00:21:01.746447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:01.746403 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:02.201509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:02.201414 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:21:02.203305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:02.203279 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:21:02.399515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:02.399469 6208 patch_prober.go:29] interesting pod/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 00:21:02.399829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:02.399792 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 00:21:02.617996 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:02.617953 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:03.375559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:03.375462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:03.387325 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:03.387291 6208 patch_prober.go:29] interesting pod/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 00:21:03.387532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:03.387366 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 00:21:04.134827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:04.134780 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:04.391974 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:04.391897 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:21:04.392466 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:04.392317 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:21:04.393124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:04.393107 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:21:04.449195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:04.449163 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:21:04.853958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:04.853926 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:05.603760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:05.603727 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:05.782579 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:21:05.782541 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:21:06.526091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:06.526029 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:07.327387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:07.327351 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:08.130419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:08.130370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:09.032700 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:09.032663 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:09.832286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:09.832245 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:10.553165 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:10.553130 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:11.436723 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:11.436688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:12.178655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:12.178620 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:12.916772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:12.916737 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:13.623282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:13.623246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:14.430853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:14.430785 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:15.216721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:15.216681 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:15.932114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:15.932079 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:16.736879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:16.736838 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:17.554073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:17.554023 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:17.668058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:17.664072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:17.769260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:17.769224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:18.630449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:18.630409 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:19.443448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:19.443416 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:20.237279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:20.237240 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:21.040380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:21.040343 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:21.747534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:21.747494 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:22.522129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:22.522090 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:23.231745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:23.231706 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:24.069141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:24.069096 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:24.998055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:24.988720 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:26.099259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:26.099224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:26.340057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:26.336182 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 00:21:26.925721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:26.925684 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:27.910549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:27.910514 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:28.853214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:28.853176 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:29.764815 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:29.764783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:32.497778 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:32.497731 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:32.609053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:32.604505 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:33.339316 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:33.339277 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:34.231050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:34.231002 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:35.032614 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:35.032579 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:35.706562 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:21:35.706511 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:21:35.855303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:35.855270 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:36.621668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:36.621626 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:37.326528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:37.326485 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:38.124180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:38.124147 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:38.825220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:38.825187 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:39.547178 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:39.547142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:40.318461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:40.318424 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:41.093788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:41.093750 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:41.933327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:41.933290 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:42.112571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:42.112536 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9"] Nov 04 00:21:42.112847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:42.112816 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" containerID="cri-o://6fbc68a0b5851a14ce8caf789c4485e922742fb2104eba9fefad65f0d1c0bb47" gracePeriod=30 Nov 04 00:21:42.827170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:42.827132 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:43.037574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:43.037541 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:43.639023 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:43.638924 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:44.451251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:44.451211 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:45.164143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:45.164108 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:46.023519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:46.023478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:46.733792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:46.733759 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:47.043661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.043564 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:47.342025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.341985 6208 generic.go:356] "Generic (PLEG): container finished" podID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerID="6fbc68a0b5851a14ce8caf789c4485e922742fb2104eba9fefad65f0d1c0bb47" exitCode=0 Nov 04 00:21:47.342350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.342325 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" event={"ID":"d633bda6-7b36-4e87-b1fe-e896c5c9072c","Type":"ContainerDied","Data":"6fbc68a0b5851a14ce8caf789c4485e922742fb2104eba9fefad65f0d1c0bb47"} Nov 04 00:21:47.418685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.418649 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 04 00:21:47.461815 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.461774 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:47.477647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.477593 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-podinfo\") pod \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " Nov 04 00:21:47.477859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.477670 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-envoy\") pod \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " Nov 04 00:21:47.477859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.477724 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-data\") pod \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " Nov 04 00:21:47.477859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.477763 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-token\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-token\") pod \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " Nov 04 00:21:47.477859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.477840 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-socket\") pod \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " Nov 04 00:21:47.478795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.477945 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-kube-api-access-zknc9\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-kube-api-access-zknc9\") pod \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " Nov 04 00:21:47.478795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.477994 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istiod-ca-cert\") pod \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " Nov 04 00:21:47.478795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.478023 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-certs\") pod \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " Nov 04 00:21:47.478795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.478062 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-data" pod "d633bda6-7b36-4e87-b1fe-e896c5c9072c" (UID: "d633bda6-7b36-4e87-b1fe-e896c5c9072c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:21:47.478795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.478162 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-credential-socket\") pod \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\" (UID: \"d633bda6-7b36-4e87-b1fe-e896c5c9072c\") " Nov 04 00:21:47.478795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.478685 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-certs" pod "d633bda6-7b36-4e87-b1fe-e896c5c9072c" (UID: "d633bda6-7b36-4e87-b1fe-e896c5c9072c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:21:47.479147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.479126 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:21:47.479241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.479148 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:21:47.479327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.479309 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-credential-socket" pod "d633bda6-7b36-4e87-b1fe-e896c5c9072c" (UID: "d633bda6-7b36-4e87-b1fe-e896c5c9072c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:21:47.479756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.479720 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istiod-ca-cert" pod "d633bda6-7b36-4e87-b1fe-e896c5c9072c" (UID: "d633bda6-7b36-4e87-b1fe-e896c5c9072c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:21:47.480086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.479997 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-socket" pod "d633bda6-7b36-4e87-b1fe-e896c5c9072c" (UID: "d633bda6-7b36-4e87-b1fe-e896c5c9072c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:21:47.482180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.482140 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-envoy" pod "d633bda6-7b36-4e87-b1fe-e896c5c9072c" (UID: "d633bda6-7b36-4e87-b1fe-e896c5c9072c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:21:47.484276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.484247 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-token" pod "d633bda6-7b36-4e87-b1fe-e896c5c9072c" (UID: "d633bda6-7b36-4e87-b1fe-e896c5c9072c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:21:47.485389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.484675 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-kube-api-access-zknc9" pod "d633bda6-7b36-4e87-b1fe-e896c5c9072c" (UID: "d633bda6-7b36-4e87-b1fe-e896c5c9072c"). InnerVolumeSpecName "kube-api-access-zknc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:21:47.485921 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.485897 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-podinfo" pod "d633bda6-7b36-4e87-b1fe-e896c5c9072c" (UID: "d633bda6-7b36-4e87-b1fe-e896c5c9072c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:21:47.591110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.586172 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zknc9\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-kube-api-access-zknc9\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:21:47.591110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.586224 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:21:47.591110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.586239 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:21:47.591110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.586252 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:21:47.591110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.586265 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:21:47.591110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.586277 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d633bda6-7b36-4e87-b1fe-e896c5c9072c-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:21:47.591110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:47.586289 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d633bda6-7b36-4e87-b1fe-e896c5c9072c-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:21:48.462057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:48.455915 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" event={"ID":"d633bda6-7b36-4e87-b1fe-e896c5c9072c","Type":"ContainerDied","Data":"0db13e03138001200fb539ee0f97eca16980ab2bec3450b7713f90cd57bb5275"} Nov 04 00:21:48.462057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:48.455994 6208 scope.go:122] "RemoveContainer" containerID="6fbc68a0b5851a14ce8caf789c4485e922742fb2104eba9fefad65f0d1c0bb47" Nov 04 00:21:48.462785 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:48.462757 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9" Nov 04 00:21:48.531577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:48.514825 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:48.622869 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:48.619798 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9"] Nov 04 00:21:48.624103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:48.624070 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f6f292-openshift-default-7d89db7dc-2vcv9"] Nov 04 00:21:49.297159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:49.297126 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" path="/var/lib/kubelet/pods/d633bda6-7b36-4e87-b1fe-e896c5c9072c/volumes" Nov 04 00:21:49.451902 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:49.451868 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:50.219158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:50.219112 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:50.864579 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:50.864540 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:51.040229 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:51.040192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:51.952720 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:51.952666 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:52.832585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:52.832549 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:52.985276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:52.985235 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77"] Nov 04 00:21:52.993816 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:52.993780 6208 cpu_manager.go:414] "containerMap: removing container" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" Nov 04 00:21:52.994076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:52.994055 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" Nov 04 00:21:52.994472 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:52.994452 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="d633bda6-7b36-4e87-b1fe-e896c5c9072c" containerName="istio-proxy" Nov 04 00:21:53.006356 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.006318 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.007920 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.007596 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77"] Nov 04 00:21:53.011088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.011044 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ecd46b\"/\"default-dockercfg-x4lzl\"" Nov 04 00:21:53.012337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.011446 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ecd46b\"/\"kube-root-ca.crt\"" Nov 04 00:21:53.012709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.012687 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ecd46b\"/\"openshift-service-ca.crt\"" Nov 04 00:21:53.012818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.012803 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ecd46b\"/\"test-model-ecd46b-kserve-self-signed-certs\"" Nov 04 00:21:53.032359 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.032297 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:53.048087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.047973 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-dshm\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.048251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.048120 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-home\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.048251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.048161 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-model-cache\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.048251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.048181 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwzt\" (UniqueName: \"kubernetes.io/projected/f73de605-a203-49e3-9401-7419f9effa52-kube-api-access-dzwzt\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.048456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.048273 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f73de605-a203-49e3-9401-7419f9effa52-tls-certs\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.048456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.048349 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-tmp-dir\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.202120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.196990 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-dshm\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.202120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.197107 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-home\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.202120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.197146 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-model-cache\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.202120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.197174 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwzt\" (UniqueName: \"kubernetes.io/projected/f73de605-a203-49e3-9401-7419f9effa52-kube-api-access-dzwzt\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.202120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.197309 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f73de605-a203-49e3-9401-7419f9effa52-tls-certs\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.202120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.197445 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-tmp-dir\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.202120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.198084 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-tmp-dir\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.202120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.200534 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-model-cache\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.202120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.200781 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-home\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.256066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.247721 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f73de605-a203-49e3-9401-7419f9effa52-tls-certs\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.256066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.251278 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwzt\" (UniqueName: \"kubernetes.io/projected/f73de605-a203-49e3-9401-7419f9effa52-kube-api-access-dzwzt\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.296509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.296457 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-dshm\") pod \"test-model-ecd46b-kserve-75bb8c799f-r5f77\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.337056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.328367 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:53.687739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.687690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:53.688333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.688309 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77"] Nov 04 00:21:53.689014 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:21:53.688982 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf73de605_a203_49e3_9401_7419f9effa52.slice/crio-f872c4386a2dc8e39936a1373483ae50c627c57920aef83248e6aa0134b2b9e6 WatchSource:0}: Error finding container f872c4386a2dc8e39936a1373483ae50c627c57920aef83248e6aa0134b2b9e6: Status 404 returned error can't find the container with id f872c4386a2dc8e39936a1373483ae50c627c57920aef83248e6aa0134b2b9e6 Nov 04 00:21:53.803061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:53.782732 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:54.514843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:54.514806 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" event={"ID":"f73de605-a203-49e3-9401-7419f9effa52","Type":"ContainerStarted","Data":"d161458d2a6bef68e629948d6d7e1072e3c876a13e27a334ad53e1929f6dc189"} Nov 04 00:21:54.515280 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:54.514852 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" event={"ID":"f73de605-a203-49e3-9401-7419f9effa52","Type":"ContainerStarted","Data":"f872c4386a2dc8e39936a1373483ae50c627c57920aef83248e6aa0134b2b9e6"} Nov 04 00:21:54.537344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:54.537287 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" podStartSLOduration=2.537271508 podStartE2EDuration="2.537271508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:21:54.533927617 +0000 UTC m=+3133.356523780" watchObservedRunningTime="2026-11-04 00:21:54.537271508 +0000 UTC m=+3133.359867670" Nov 04 00:21:54.543234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:54.543201 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:55.254675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:55.254639 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:55.378066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:55.372264 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:21:56.129064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:56.129011 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:57.038624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.038588 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:57.260138 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.260098 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74"] Nov 04 00:21:57.265433 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.265388 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.277408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.277379 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9a6f7a-openshift-default-dockercfg-m7fs7\"" Nov 04 00:21:57.279551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.279525 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74"] Nov 04 00:21:57.404657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.404561 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-credential-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.404657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.404603 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-data\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.404857 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.404782 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.404857 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.404822 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-token\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.404926 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.404911 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksffc\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-kube-api-access-ksffc\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.404979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.404966 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-certs\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.405161 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.405144 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-envoy\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.405797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.405171 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-podinfo\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.405797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.405193 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istiod-ca-cert\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.506999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.506951 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.506999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.506998 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-token\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.507291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507027 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksffc\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-kube-api-access-ksffc\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.507291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507093 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-certs\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.507291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507240 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-envoy\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.507291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507266 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-podinfo\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.507291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507289 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istiod-ca-cert\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.507556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507408 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.507556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507438 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-certs\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.507556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507454 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-credential-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.507556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507488 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-data\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.508852 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507686 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-credential-socket\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.508852 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507877 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-data\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.508852 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.507980 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istiod-ca-cert\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.509642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.509620 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-envoy\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.509903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.509867 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-podinfo\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.519233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.519210 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-token\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.519381 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.519363 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksffc\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-kube-api-access-ksffc\") pod \"e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.587183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.587143 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:57.888395 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.888358 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74"] Nov 04 00:21:57.889252 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:21:57.889222 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e32687_55f2_4eb8_bf9d_af860aa9848e.slice/crio-b5b99398e864262ac7f6fe8e493850e8cd1c0b047c2d043078bfa1c69523d8cc WatchSource:0}: Error finding container b5b99398e864262ac7f6fe8e493850e8cd1c0b047c2d043078bfa1c69523d8cc: Status 404 returned error can't find the container with id b5b99398e864262ac7f6fe8e493850e8cd1c0b047c2d043078bfa1c69523d8cc Nov 04 00:21:57.892754 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.892716 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:21:57.892859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.892810 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:21:57.892926 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:57.892859 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:21:58.019946 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:58.019914 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:58.549774 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:58.549732 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" event={"ID":"e8e32687-55f2-4eb8-bf9d-af860aa9848e","Type":"ContainerStarted","Data":"f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8"} Nov 04 00:21:58.550294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:58.549791 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" event={"ID":"e8e32687-55f2-4eb8-bf9d-af860aa9848e","Type":"ContainerStarted","Data":"b5b99398e864262ac7f6fe8e493850e8cd1c0b047c2d043078bfa1c69523d8cc"} Nov 04 00:21:58.576773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:58.576720 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podStartSLOduration=1.5767058010000001 podStartE2EDuration="1.576705801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:21:58.575202553 +0000 UTC m=+3137.397798718" watchObservedRunningTime="2026-11-04 00:21:58.576705801 +0000 UTC m=+3137.399301963" Nov 04 00:21:58.588283 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:58.588248 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:21:58.589868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:58.589843 6208 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 00:21:58.589980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:58.589890 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 00:21:58.752532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:58.752489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:21:59.587794 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:59.587757 6208 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 00:21:59.588247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:59.587816 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 00:21:59.642940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:21:59.642900 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:00.523945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:00.523906 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:00.596187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:00.596134 6208 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 00:22:00.600326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:00.600285 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 00:22:01.234158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:01.234104 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:01.588329 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:01.588243 6208 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 00:22:01.588329 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:01.588321 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 00:22:02.050840 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:02.050798 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:02.588597 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:02.588557 6208 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 00:22:02.588814 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:02.588613 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 00:22:02.837318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:02.837247 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:03.329702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:03.329658 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:22:03.342855 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:03.342820 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:22:03.591980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:03.591880 6208 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 00:22:03.592287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:03.591968 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 00:22:03.703684 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:03.703643 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:22:03.732747 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:03.732683 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:04.588576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:04.588540 6208 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 04 00:22:04.589012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:04.588642 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 04 00:22:04.615764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:04.615731 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:05.364682 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:05.364646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:05.593078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:05.593013 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:22:05.593562 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:05.593524 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:22:05.594181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:05.594156 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:22:05.723307 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:22:05.723259 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:22:06.167435 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:06.167349 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:06.272151 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:06.272108 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:07.189398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:07.189363 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:07.924238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:07.924203 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:08.722029 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:08.721995 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:09.425796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:09.425759 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:10.227790 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:10.227751 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:10.995366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:10.995325 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:11.688229 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:11.688189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:12.506007 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:12.505970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:13.303173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:13.303138 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:14.020020 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:14.019986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:14.318543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:14.318426 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:15.027134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:15.027099 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:15.910333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:15.910301 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:16.975212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:16.975170 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:17.953795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:17.953749 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:19.041664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:19.041626 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:19.946553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:19.946464 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:20.845924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:20.845882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:21.837839 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:21.837803 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:22.866570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:22.866535 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:23.988324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:23.988289 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:25.064404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:25.064361 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:26.034680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:26.034641 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:27.008514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:27.008471 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:27.131113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:27.128641 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:27.612800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:27.612761 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:28.032347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:28.032308 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:28.832768 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:28.832733 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:29.805997 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:29.805956 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:31.001617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:31.001547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:32.340715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:32.340679 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:33.255354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:33.255309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:34.457890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:34.457854 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:35.165690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:35.165629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:35.265663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:35.265621 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:35.394168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:35.394129 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:35.963726 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:22:35.963054 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:22:36.278823 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:36.278731 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:37.250238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:37.250205 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:37.746951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:37.746917 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:38.074981 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:38.074886 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:38.994131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:38.994085 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:40.008717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:40.008683 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:40.933715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:40.933679 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:42.015394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:42.012166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:43.009661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:43.009620 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:43.119054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:43.117485 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:43.741779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:43.741747 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:44.605343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:44.605304 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:44.726056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:44.721884 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:45.089967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:45.089925 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:45.498870 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:45.498834 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:45.596084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:45.596013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:46.318230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:46.318145 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:46.440169 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:46.440131 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:47.104495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:47.104454 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:47.225787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:47.225651 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:52.122766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:52.122715 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:52.224475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:52.224441 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:52.744416 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:52.744379 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:53.742818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:53.742782 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:22:58.815162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:22:58.815121 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:05.710733 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:05.710688 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:23:13.640831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:13.640787 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:15.435238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:15.435201 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:18.305847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:18.305809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:19.422738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:19.422698 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:20.193575 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:20.193538 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:21.089498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:21.089464 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:21.458446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:21.458412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:22.143665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:22.143626 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:22.957579 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:22.957539 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:23.826200 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:23.826167 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:23.931571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:23.931536 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:24.432357 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:24.432322 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m"] Nov 04 00:23:24.432820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:24.432786 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" podUID="71e0e69d-ea9d-4bf1-b4cb-14c10731a974" containerName="maas-api" containerID="cri-o://0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5" gracePeriod=30 Nov 04 00:23:24.479569 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:24.479539 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:25.058181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.054844 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:25.125731 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.125352 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:23:25.169920 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.169854 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-metrics-certs\") pod \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " Nov 04 00:23:25.170346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.170312 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-maas-api-tls\") pod \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " Nov 04 00:23:25.171149 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.171126 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-openshift-service-ca\") pod \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " Nov 04 00:23:25.201092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.175125 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-kube-api-access-9wqlh\" (UniqueName: \"kubernetes.io/projected/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-kube-api-access-9wqlh\") pod \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\" (UID: \"71e0e69d-ea9d-4bf1-b4cb-14c10731a974\") " Nov 04 00:23:25.201092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.179932 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-kube-api-access-9wqlh" pod "71e0e69d-ea9d-4bf1-b4cb-14c10731a974" (UID: "71e0e69d-ea9d-4bf1-b4cb-14c10731a974"). InnerVolumeSpecName "kube-api-access-9wqlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:25.201092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.186247 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-openshift-service-ca" pod "71e0e69d-ea9d-4bf1-b4cb-14c10731a974" (UID: "71e0e69d-ea9d-4bf1-b4cb-14c10731a974"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:23:25.201092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.190500 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-maas-api-tls" pod "71e0e69d-ea9d-4bf1-b4cb-14c10731a974" (UID: "71e0e69d-ea9d-4bf1-b4cb-14c10731a974"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:23:25.209517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.209484 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-metrics-certs" pod "71e0e69d-ea9d-4bf1-b4cb-14c10731a974" (UID: "71e0e69d-ea9d-4bf1-b4cb-14c10731a974"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:23:25.278011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.277918 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:25.278011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.277952 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:25.278011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.277967 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:25.278011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.277977 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqlh\" (UniqueName: \"kubernetes.io/projected/71e0e69d-ea9d-4bf1-b4cb-14c10731a974-kube-api-access-9wqlh\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:25.285808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.285781 6208 generic.go:356] "Generic (PLEG): container finished" podID="71e0e69d-ea9d-4bf1-b4cb-14c10731a974" containerID="0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5" exitCode=0 Nov 04 00:23:25.285944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.285829 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" Nov 04 00:23:25.285944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.285861 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" event={"ID":"71e0e69d-ea9d-4bf1-b4cb-14c10731a974","Type":"ContainerDied","Data":"0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5"} Nov 04 00:23:25.285944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.285916 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m" event={"ID":"71e0e69d-ea9d-4bf1-b4cb-14c10731a974","Type":"ContainerDied","Data":"ca90f5410c1a6f9b730b287e59cfa0a6e119996096d9fb771586fad20022c396"} Nov 04 00:23:25.285944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.285940 6208 scope.go:122] "RemoveContainer" containerID="0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5" Nov 04 00:23:25.298344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.298324 6208 scope.go:122] "RemoveContainer" containerID="0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5" Nov 04 00:23:25.298647 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:25.298617 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5\": container with ID starting with 0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5 not found: ID does not exist" containerID="0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5" Nov 04 00:23:25.298748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.298654 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5"} err="failed to get container status \"0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5\": rpc error: code = NotFound desc = could not find container \"0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5\": container with ID starting with 0e3cf198c527d81864460203288eb1764dcac9c33690b099388ac47f2b69f8f5 not found: ID does not exist" Nov 04 00:23:25.314209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.314186 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m"] Nov 04 00:23:25.318685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.318665 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8cc1c0-5f64fd4868-44w8m"] Nov 04 00:23:25.803983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:25.803949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:26.677488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:26.677454 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:27.297741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:27.297703 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e0e69d-ea9d-4bf1-b4cb-14c10731a974" path="/var/lib/kubelet/pods/71e0e69d-ea9d-4bf1-b4cb-14c10731a974/volumes" Nov 04 00:23:27.458761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:27.458731 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:27.590907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:27.590791 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6"] Nov 04 00:23:27.591137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:27.591106 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" containerID="cri-o://7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359" gracePeriod=30 Nov 04 00:23:28.280159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:28.280121 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:29.125882 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:29.125842 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:29.957860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:29.957824 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:30.734564 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:30.734523 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:31.463184 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:31.463152 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:32.252478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:32.252446 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:32.907787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:32.907760 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 04 00:23:33.053543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.053513 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/257b95e5-b863-4444-9a35-efd3ca15e157-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257b95e5-b863-4444-9a35-efd3ca15e157-istio-podinfo\") pod \"257b95e5-b863-4444-9a35-efd3ca15e157\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " Nov 04 00:23:33.053755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.053559 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-data\") pod \"257b95e5-b863-4444-9a35-efd3ca15e157\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " Nov 04 00:23:33.053755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.053581 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-credential-socket\") pod \"257b95e5-b863-4444-9a35-efd3ca15e157\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " Nov 04 00:23:33.053755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.053663 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-socket\") pod \"257b95e5-b863-4444-9a35-efd3ca15e157\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " Nov 04 00:23:33.053932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.053880 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-certs\") pod \"257b95e5-b863-4444-9a35-efd3ca15e157\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " Nov 04 00:23:33.053998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.053966 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257b95e5-b863-4444-9a35-efd3ca15e157-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257b95e5-b863-4444-9a35-efd3ca15e157-istiod-ca-cert\") pod \"257b95e5-b863-4444-9a35-efd3ca15e157\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " Nov 04 00:23:33.053998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.053976 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-socket" pod "257b95e5-b863-4444-9a35-efd3ca15e157" (UID: "257b95e5-b863-4444-9a35-efd3ca15e157"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:33.054114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.053940 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-data" pod "257b95e5-b863-4444-9a35-efd3ca15e157" (UID: "257b95e5-b863-4444-9a35-efd3ca15e157"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:33.054114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.053995 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-credential-socket" pod "257b95e5-b863-4444-9a35-efd3ca15e157" (UID: "257b95e5-b863-4444-9a35-efd3ca15e157"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:33.054114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.054092 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-istio-token\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-istio-token\") pod \"257b95e5-b863-4444-9a35-efd3ca15e157\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " Nov 04 00:23:33.054274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.054171 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-certs" pod "257b95e5-b863-4444-9a35-efd3ca15e157" (UID: "257b95e5-b863-4444-9a35-efd3ca15e157"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:33.054274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.054194 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-kube-api-access-vblj4\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-kube-api-access-vblj4\") pod \"257b95e5-b863-4444-9a35-efd3ca15e157\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " Nov 04 00:23:33.054351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.054298 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-envoy\") pod \"257b95e5-b863-4444-9a35-efd3ca15e157\" (UID: \"257b95e5-b863-4444-9a35-efd3ca15e157\") " Nov 04 00:23:33.054400 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.054339 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257b95e5-b863-4444-9a35-efd3ca15e157-istiod-ca-cert" pod "257b95e5-b863-4444-9a35-efd3ca15e157" (UID: "257b95e5-b863-4444-9a35-efd3ca15e157"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:23:33.055015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.054987 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:33.055158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.055019 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/257b95e5-b863-4444-9a35-efd3ca15e157-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:33.055158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.055050 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:33.055158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.055066 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:33.055158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.055081 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:33.055827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.055800 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/257b95e5-b863-4444-9a35-efd3ca15e157-istio-podinfo" pod "257b95e5-b863-4444-9a35-efd3ca15e157" (UID: "257b95e5-b863-4444-9a35-efd3ca15e157"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:23:33.056741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.056721 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-istio-token" pod "257b95e5-b863-4444-9a35-efd3ca15e157" (UID: "257b95e5-b863-4444-9a35-efd3ca15e157"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:33.057132 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.057112 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-kube-api-access-vblj4" pod "257b95e5-b863-4444-9a35-efd3ca15e157" (UID: "257b95e5-b863-4444-9a35-efd3ca15e157"). InnerVolumeSpecName "kube-api-access-vblj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:33.057185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.057128 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-envoy" pod "257b95e5-b863-4444-9a35-efd3ca15e157" (UID: "257b95e5-b863-4444-9a35-efd3ca15e157"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:33.064296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.064262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:33.175525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.175391 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:33.175525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.175436 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vblj4\" (UniqueName: \"kubernetes.io/projected/257b95e5-b863-4444-9a35-efd3ca15e157-kube-api-access-vblj4\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:33.175525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.175470 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/257b95e5-b863-4444-9a35-efd3ca15e157-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:33.175525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.175484 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/257b95e5-b863-4444-9a35-efd3ca15e157-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:33.389652 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.389621 6208 generic.go:356] "Generic (PLEG): container finished" podID="257b95e5-b863-4444-9a35-efd3ca15e157" containerID="7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359" exitCode=0 Nov 04 00:23:33.389812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.389676 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" Nov 04 00:23:33.389812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.389692 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" event={"ID":"257b95e5-b863-4444-9a35-efd3ca15e157","Type":"ContainerDied","Data":"7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359"} Nov 04 00:23:33.389812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.389750 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6" event={"ID":"257b95e5-b863-4444-9a35-efd3ca15e157","Type":"ContainerDied","Data":"9df4c97f299bce5f2e1771b1b4ac681f9fd520f5f880fccc2b2477a9932ec3e8"} Nov 04 00:23:33.389812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.389777 6208 scope.go:122] "RemoveContainer" containerID="7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359" Nov 04 00:23:33.406655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.405996 6208 scope.go:122] "RemoveContainer" containerID="6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e" Nov 04 00:23:33.413792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.413769 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6"] Nov 04 00:23:33.420190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.420172 6208 scope.go:122] "RemoveContainer" containerID="7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359" Nov 04 00:23:33.420455 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:33.420432 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359\": container with ID starting with 7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359 not found: ID does not exist" containerID="7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359" Nov 04 00:23:33.420561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.420461 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359"} err="failed to get container status \"7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359\": rpc error: code = NotFound desc = could not find container \"7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359\": container with ID starting with 7055f0b47c7b7ca0881a95eb64f1ba68fab0f711661baca188942e175589c359 not found: ID does not exist" Nov 04 00:23:33.420561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.420478 6208 scope.go:122] "RemoveContainer" containerID="6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e" Nov 04 00:23:33.420755 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:33.420735 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e\": container with ID starting with 6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e not found: ID does not exist" containerID="6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e" Nov 04 00:23:33.420814 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.420761 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e"} err="failed to get container status \"6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e\": rpc error: code = NotFound desc = could not find container \"6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e\": container with ID starting with 6f002147988549a65c1b67e891d7e4ccd25958943ecd31f57e31a54ddafad70e not found: ID does not exist" Nov 04 00:23:33.422534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.422512 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a5948-openshift-default-6867d848f-xmpt6"] Nov 04 00:23:33.907537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:33.907504 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:34.751154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:34.751070 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:34.862074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:34.861813 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:35.298106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.298075 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" path="/var/lib/kubelet/pods/257b95e5-b863-4444-9a35-efd3ca15e157/volumes" Nov 04 00:23:35.314411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.314374 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:35.707950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.707826 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:35.716992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.716955 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b"] Nov 04 00:23:35.717514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.717447 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" containerID="cri-o://fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b" gracePeriod=30 Nov 04 00:23:35.735957 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.735933 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj"] Nov 04 00:23:35.736232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.736185 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" containerID="cri-o://cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60" gracePeriod=30 Nov 04 00:23:35.741735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.741713 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv"] Nov 04 00:23:35.742601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.741922 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" containerID="cri-o://49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1" gracePeriod=30 Nov 04 00:23:35.825062 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:35.817331 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:23:35.971132 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:35.962788 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:36.835244 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:36.835210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:37.559824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:37.559791 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:38.329271 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:38.329241 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:38.523388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:38.523347 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:23:38.523570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:38.523408 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:23:39.054412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:39.054379 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:39.824317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:39.824285 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:40.227000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:40.226965 6208 patch_prober.go:29] interesting pod/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:23:40.227217 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:40.227056 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:23:40.604235 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:40.604149 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:41.104612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.104570 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:23:41.106907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.106883 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:23:41.176228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176186 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-token\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-token\") pod \"2d797191-b9e1-4fc7-88bb-4eda431424b4\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " Nov 04 00:23:41.176410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176301 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d797191-b9e1-4fc7-88bb-4eda431424b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d797191-b9e1-4fc7-88bb-4eda431424b4-istiod-ca-cert\") pod \"2d797191-b9e1-4fc7-88bb-4eda431424b4\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " Nov 04 00:23:41.176410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176386 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-data\") pod \"2d797191-b9e1-4fc7-88bb-4eda431424b4\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " Nov 04 00:23:41.176550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176456 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-socket\") pod \"4526c8ef-a23a-492b-99cf-31957ea71f0f\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " Nov 04 00:23:41.176550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176499 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-kube-api-access-22k97\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-kube-api-access-22k97\") pod \"2d797191-b9e1-4fc7-88bb-4eda431424b4\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " Nov 04 00:23:41.176671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176577 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-podinfo\") pod \"4526c8ef-a23a-492b-99cf-31957ea71f0f\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " Nov 04 00:23:41.176671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176621 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-socket\") pod \"2d797191-b9e1-4fc7-88bb-4eda431424b4\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " Nov 04 00:23:41.176787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176679 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-certs\") pod \"2d797191-b9e1-4fc7-88bb-4eda431424b4\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " Nov 04 00:23:41.176787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176694 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:23:41.176787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176773 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-credential-socket\") pod \"2d797191-b9e1-4fc7-88bb-4eda431424b4\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " Nov 04 00:23:41.176951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176825 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-certs\") pod \"4526c8ef-a23a-492b-99cf-31957ea71f0f\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " Nov 04 00:23:41.176951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176882 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-data\") pod \"4526c8ef-a23a-492b-99cf-31957ea71f0f\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " Nov 04 00:23:41.176951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176916 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-kube-api-access-zz2rq\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-kube-api-access-zz2rq\") pod \"4526c8ef-a23a-492b-99cf-31957ea71f0f\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " Nov 04 00:23:41.177147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.176976 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-token\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-token\") pod \"4526c8ef-a23a-492b-99cf-31957ea71f0f\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " Nov 04 00:23:41.177147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.177005 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-envoy\") pod \"4526c8ef-a23a-492b-99cf-31957ea71f0f\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " Nov 04 00:23:41.177147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.177132 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-podinfo\") pod \"2d797191-b9e1-4fc7-88bb-4eda431424b4\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " Nov 04 00:23:41.177312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.177191 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4526c8ef-a23a-492b-99cf-31957ea71f0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4526c8ef-a23a-492b-99cf-31957ea71f0f-istiod-ca-cert\") pod \"4526c8ef-a23a-492b-99cf-31957ea71f0f\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " Nov 04 00:23:41.177312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.177228 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-credential-socket\") pod \"4526c8ef-a23a-492b-99cf-31957ea71f0f\" (UID: \"4526c8ef-a23a-492b-99cf-31957ea71f0f\") " Nov 04 00:23:41.177312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.177270 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-envoy\") pod \"2d797191-b9e1-4fc7-88bb-4eda431424b4\" (UID: \"2d797191-b9e1-4fc7-88bb-4eda431424b4\") " Nov 04 00:23:41.177473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.177372 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-certs" pod "4526c8ef-a23a-492b-99cf-31957ea71f0f" (UID: "4526c8ef-a23a-492b-99cf-31957ea71f0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.177578 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.177556 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-socket" pod "2d797191-b9e1-4fc7-88bb-4eda431424b4" (UID: "2d797191-b9e1-4fc7-88bb-4eda431424b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.178258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.177723 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-certs" pod "2d797191-b9e1-4fc7-88bb-4eda431424b4" (UID: "2d797191-b9e1-4fc7-88bb-4eda431424b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.178350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.178281 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-data" pod "2d797191-b9e1-4fc7-88bb-4eda431424b4" (UID: "2d797191-b9e1-4fc7-88bb-4eda431424b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.179176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.178489 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-credential-socket" pod "2d797191-b9e1-4fc7-88bb-4eda431424b4" (UID: "2d797191-b9e1-4fc7-88bb-4eda431424b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.179176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.178855 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-socket" pod "4526c8ef-a23a-492b-99cf-31957ea71f0f" (UID: "4526c8ef-a23a-492b-99cf-31957ea71f0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.179176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.178801 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d797191-b9e1-4fc7-88bb-4eda431424b4-istiod-ca-cert" pod "2d797191-b9e1-4fc7-88bb-4eda431424b4" (UID: "2d797191-b9e1-4fc7-88bb-4eda431424b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:23:41.179382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.179224 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.179382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.179244 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.179382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.179258 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.179382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.179271 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.179382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.179286 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.179658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.179399 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-data" pod "4526c8ef-a23a-492b-99cf-31957ea71f0f" (UID: "4526c8ef-a23a-492b-99cf-31957ea71f0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.179658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.179327 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4526c8ef-a23a-492b-99cf-31957ea71f0f-istiod-ca-cert" pod "4526c8ef-a23a-492b-99cf-31957ea71f0f" (UID: "4526c8ef-a23a-492b-99cf-31957ea71f0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:23:41.179658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.179479 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-credential-socket" pod "4526c8ef-a23a-492b-99cf-31957ea71f0f" (UID: "4526c8ef-a23a-492b-99cf-31957ea71f0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.181500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.180503 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-token" pod "2d797191-b9e1-4fc7-88bb-4eda431424b4" (UID: "2d797191-b9e1-4fc7-88bb-4eda431424b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:41.181500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.181465 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-podinfo" pod "4526c8ef-a23a-492b-99cf-31957ea71f0f" (UID: "4526c8ef-a23a-492b-99cf-31957ea71f0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:23:41.182083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.182057 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-podinfo" pod "2d797191-b9e1-4fc7-88bb-4eda431424b4" (UID: "2d797191-b9e1-4fc7-88bb-4eda431424b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:23:41.182667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.182565 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-kube-api-access-zz2rq" pod "4526c8ef-a23a-492b-99cf-31957ea71f0f" (UID: "4526c8ef-a23a-492b-99cf-31957ea71f0f"). InnerVolumeSpecName "kube-api-access-zz2rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:41.183624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.183602 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-token" pod "4526c8ef-a23a-492b-99cf-31957ea71f0f" (UID: "4526c8ef-a23a-492b-99cf-31957ea71f0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:41.184419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.184384 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-envoy" pod "2d797191-b9e1-4fc7-88bb-4eda431424b4" (UID: "2d797191-b9e1-4fc7-88bb-4eda431424b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.184521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.184428 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-kube-api-access-22k97" pod "2d797191-b9e1-4fc7-88bb-4eda431424b4" (UID: "2d797191-b9e1-4fc7-88bb-4eda431424b4"). InnerVolumeSpecName "kube-api-access-22k97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:41.185557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.185531 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-envoy" pod "4526c8ef-a23a-492b-99cf-31957ea71f0f" (UID: "4526c8ef-a23a-492b-99cf-31957ea71f0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.280815 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.280784 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-certs\") pod \"1156db56-b84d-4241-9689-69a6aa1eaf5e\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " Nov 04 00:23:41.281009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.280844 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-kube-api-access-t2mm9\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-kube-api-access-t2mm9\") pod \"1156db56-b84d-4241-9689-69a6aa1eaf5e\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " Nov 04 00:23:41.281009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.280951 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-data\") pod \"1156db56-b84d-4241-9689-69a6aa1eaf5e\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " Nov 04 00:23:41.281009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.280982 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-socket\") pod \"1156db56-b84d-4241-9689-69a6aa1eaf5e\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " Nov 04 00:23:41.281224 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281108 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-envoy\") pod \"1156db56-b84d-4241-9689-69a6aa1eaf5e\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " Nov 04 00:23:41.281224 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281162 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-token\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-token\") pod \"1156db56-b84d-4241-9689-69a6aa1eaf5e\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " Nov 04 00:23:41.281224 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281163 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-certs" pod "1156db56-b84d-4241-9689-69a6aa1eaf5e" (UID: "1156db56-b84d-4241-9689-69a6aa1eaf5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.281224 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281206 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-podinfo\") pod \"1156db56-b84d-4241-9689-69a6aa1eaf5e\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " Nov 04 00:23:41.281504 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281340 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-data" pod "1156db56-b84d-4241-9689-69a6aa1eaf5e" (UID: "1156db56-b84d-4241-9689-69a6aa1eaf5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.281504 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281356 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-socket" pod "1156db56-b84d-4241-9689-69a6aa1eaf5e" (UID: "1156db56-b84d-4241-9689-69a6aa1eaf5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.281504 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281467 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-credential-socket\") pod \"1156db56-b84d-4241-9689-69a6aa1eaf5e\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " Nov 04 00:23:41.281680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281533 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1156db56-b84d-4241-9689-69a6aa1eaf5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156db56-b84d-4241-9689-69a6aa1eaf5e-istiod-ca-cert\") pod \"1156db56-b84d-4241-9689-69a6aa1eaf5e\" (UID: \"1156db56-b84d-4241-9689-69a6aa1eaf5e\") " Nov 04 00:23:41.281859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281803 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-credential-socket" pod "1156db56-b84d-4241-9689-69a6aa1eaf5e" (UID: "1156db56-b84d-4241-9689-69a6aa1eaf5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.282023 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.281997 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1156db56-b84d-4241-9689-69a6aa1eaf5e-istiod-ca-cert" pod "1156db56-b84d-4241-9689-69a6aa1eaf5e" (UID: "1156db56-b84d-4241-9689-69a6aa1eaf5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:23:41.282694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282670 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282703 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4526c8ef-a23a-492b-99cf-31957ea71f0f-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282719 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282734 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282748 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1156db56-b84d-4241-9689-69a6aa1eaf5e-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282762 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282775 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d797191-b9e1-4fc7-88bb-4eda431424b4-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282790 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282806 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282822 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22k97\" (UniqueName: \"kubernetes.io/projected/2d797191-b9e1-4fc7-88bb-4eda431424b4-kube-api-access-22k97\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.282848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282837 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.283344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282852 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.283344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282867 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.283344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282882 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.283344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282898 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz2rq\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-kube-api-access-zz2rq\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.283344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282912 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.283344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282926 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4526c8ef-a23a-492b-99cf-31957ea71f0f-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.283344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.282940 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d797191-b9e1-4fc7-88bb-4eda431424b4-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.283782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.283618 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-envoy" pod "1156db56-b84d-4241-9689-69a6aa1eaf5e" (UID: "1156db56-b84d-4241-9689-69a6aa1eaf5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:41.283782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.283736 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-token" pod "1156db56-b84d-4241-9689-69a6aa1eaf5e" (UID: "1156db56-b84d-4241-9689-69a6aa1eaf5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:41.283782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.283760 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-kube-api-access-t2mm9" pod "1156db56-b84d-4241-9689-69a6aa1eaf5e" (UID: "1156db56-b84d-4241-9689-69a6aa1eaf5e"). InnerVolumeSpecName "kube-api-access-t2mm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:41.284163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.284143 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-podinfo" pod "1156db56-b84d-4241-9689-69a6aa1eaf5e" (UID: "1156db56-b84d-4241-9689-69a6aa1eaf5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:23:41.355120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.355086 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:41.385504 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.385482 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2mm9\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-kube-api-access-t2mm9\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.385504 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.385504 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.385710 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.385517 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.385710 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.385526 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1156db56-b84d-4241-9689-69a6aa1eaf5e-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:41.488559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.488517 6208 generic.go:356] "Generic (PLEG): container finished" podID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerID="fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b" exitCode=0 Nov 04 00:23:41.489667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.488567 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" Nov 04 00:23:41.489667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.488601 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" event={"ID":"4526c8ef-a23a-492b-99cf-31957ea71f0f","Type":"ContainerDied","Data":"fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b"} Nov 04 00:23:41.489667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.488666 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b" event={"ID":"4526c8ef-a23a-492b-99cf-31957ea71f0f","Type":"ContainerDied","Data":"75dc8e15406b51ff7417161d6c5168e217f0b1646d94b7bfd79f1d0b6150d2d9"} Nov 04 00:23:41.489667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.488688 6208 scope.go:122] "RemoveContainer" containerID="fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b" Nov 04 00:23:41.490641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.490620 6208 generic.go:356] "Generic (PLEG): container finished" podID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerID="cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60" exitCode=0 Nov 04 00:23:41.490721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.490684 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" Nov 04 00:23:41.491022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.490714 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" event={"ID":"1156db56-b84d-4241-9689-69a6aa1eaf5e","Type":"ContainerDied","Data":"cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60"} Nov 04 00:23:41.491022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.490769 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj" event={"ID":"1156db56-b84d-4241-9689-69a6aa1eaf5e","Type":"ContainerDied","Data":"7d1f8097eea981a6fbe4306e827ff1c4a8e4635835fe4beecc561784b2801cfb"} Nov 04 00:23:41.492642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.492620 6208 generic.go:356] "Generic (PLEG): container finished" podID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerID="49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1" exitCode=0 Nov 04 00:23:41.492752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.492656 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" event={"ID":"2d797191-b9e1-4fc7-88bb-4eda431424b4","Type":"ContainerDied","Data":"49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1"} Nov 04 00:23:41.492752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.492684 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" Nov 04 00:23:41.492752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.492705 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv" event={"ID":"2d797191-b9e1-4fc7-88bb-4eda431424b4","Type":"ContainerDied","Data":"252e8bb6c272207752281416088f877dfda643fefea0745ddb7911ee40114af1"} Nov 04 00:23:41.503629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.503606 6208 scope.go:122] "RemoveContainer" containerID="fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b" Nov 04 00:23:41.503992 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:41.503925 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b\": container with ID starting with fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b not found: ID does not exist" containerID="fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b" Nov 04 00:23:41.503992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.503954 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b"} err="failed to get container status \"fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b\": rpc error: code = NotFound desc = could not find container \"fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b\": container with ID starting with fd5ad66962482a6feae4646ff40b8840e3d1725d0c32d09918224edffda93a5b not found: ID does not exist" Nov 04 00:23:41.503992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.503970 6208 scope.go:122] "RemoveContainer" containerID="cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60" Nov 04 00:23:41.516944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.516928 6208 scope.go:122] "RemoveContainer" containerID="48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689" Nov 04 00:23:41.521986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.521937 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b"] Nov 04 00:23:41.529832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.529801 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-2p77b"] Nov 04 00:23:41.530602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.530584 6208 scope.go:122] "RemoveContainer" containerID="cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60" Nov 04 00:23:41.530904 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:41.530883 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60\": container with ID starting with cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60 not found: ID does not exist" containerID="cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60" Nov 04 00:23:41.530975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.530912 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60"} err="failed to get container status \"cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60\": rpc error: code = NotFound desc = could not find container \"cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60\": container with ID starting with cc817f6e9c827820b659104f29d158253f3adcfc3bfb8876016c267ea6bb0b60 not found: ID does not exist" Nov 04 00:23:41.530975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.530929 6208 scope.go:122] "RemoveContainer" containerID="48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689" Nov 04 00:23:41.531221 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:41.531193 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689\": container with ID starting with 48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689 not found: ID does not exist" containerID="48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689" Nov 04 00:23:41.531267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.531230 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689"} err="failed to get container status \"48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689\": rpc error: code = NotFound desc = could not find container \"48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689\": container with ID starting with 48aaec218240575b8c1567289322fe03fb01562b65c08cb2e60bf27d162f1689 not found: ID does not exist" Nov 04 00:23:41.531267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.531244 6208 scope.go:122] "RemoveContainer" containerID="49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1" Nov 04 00:23:41.543096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.543069 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv"] Nov 04 00:23:41.556074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.545101 6208 scope.go:122] "RemoveContainer" containerID="49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1" Nov 04 00:23:41.556074 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:41.547362 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1\": container with ID starting with 49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1 not found: ID does not exist" containerID="49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1" Nov 04 00:23:41.556074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.547407 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1"} err="failed to get container status \"49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1\": rpc error: code = NotFound desc = could not find container \"49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1\": container with ID starting with 49dcef208dda69bd16d662715e15cf5d13047befd880f2f5eca9d277e97c6ca1 not found: ID does not exist" Nov 04 00:23:41.556074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.552684 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-7rgfv"] Nov 04 00:23:41.565882 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.565848 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj"] Nov 04 00:23:41.570663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.570640 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8cc1c0-openshift-default-678dbfbc89-pb5pj"] Nov 04 00:23:41.723013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.722979 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:41.822575 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.822546 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:41.924199 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:41.924164 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:42.172083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:42.171968 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh"] Nov 04 00:23:42.172540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:42.172263 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" containerID="cri-o://5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b" gracePeriod=30 Nov 04 00:23:42.178485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:42.178457 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq"] Nov 04 00:23:42.178670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:42.178647 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" containerID="cri-o://c5465e3d6ea30942d305d7f2260dac0b8fa21d82f1f06676fb049102ca0a6100" gracePeriod=30 Nov 04 00:23:42.325406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:42.325366 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:42.337611 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:42.337578 6208 patch_prober.go:29] interesting pod/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:23:42.337799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:42.337652 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:23:43.147940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:43.147905 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:43.280760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:43.279111 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:43.303321 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:43.303286 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" path="/var/lib/kubelet/pods/1156db56-b84d-4241-9689-69a6aa1eaf5e/volumes" Nov 04 00:23:43.303890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:43.303874 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" path="/var/lib/kubelet/pods/2d797191-b9e1-4fc7-88bb-4eda431424b4/volumes" Nov 04 00:23:43.304409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:43.304394 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" path="/var/lib/kubelet/pods/4526c8ef-a23a-492b-99cf-31957ea71f0f/volumes" Nov 04 00:23:43.957569 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:43.957535 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:44.744334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:44.744298 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:45.442159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:45.442115 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:46.229484 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:46.229453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:47.093734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.093700 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:47.210383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.205997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:47.463256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.463220 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:23:47.567372 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.567335 6208 generic.go:356] "Generic (PLEG): container finished" podID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerID="5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b" exitCode=0 Nov 04 00:23:47.567553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.567403 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" Nov 04 00:23:47.567553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.567416 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" event={"ID":"8390a996-22ca-4972-90d7-c2c6f104c4cc","Type":"ContainerDied","Data":"5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b"} Nov 04 00:23:47.567553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.567479 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh" event={"ID":"8390a996-22ca-4972-90d7-c2c6f104c4cc","Type":"ContainerDied","Data":"4b526035bfcbe1ec537f76dd39b88e408c4d3ffe9a771c8c6a2215856e7b3d5e"} Nov 04 00:23:47.567553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.567508 6208 scope.go:122] "RemoveContainer" containerID="5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b" Nov 04 00:23:47.569887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.569836 6208 generic.go:356] "Generic (PLEG): container finished" podID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerID="c5465e3d6ea30942d305d7f2260dac0b8fa21d82f1f06676fb049102ca0a6100" exitCode=0 Nov 04 00:23:47.570075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.569879 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" event={"ID":"3fb44f32-c801-4a3b-b97b-81e776ddbbd6","Type":"ContainerDied","Data":"c5465e3d6ea30942d305d7f2260dac0b8fa21d82f1f06676fb049102ca0a6100"} Nov 04 00:23:47.579617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.579589 6208 scope.go:122] "RemoveContainer" containerID="5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b" Nov 04 00:23:47.579901 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:23:47.579880 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b\": container with ID starting with 5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b not found: ID does not exist" containerID="5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b" Nov 04 00:23:47.579955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.579908 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b"} err="failed to get container status \"5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b\": rpc error: code = NotFound desc = could not find container \"5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b\": container with ID starting with 5243f00585410f734ceab0f5318598c858269f53b6867ad0f9787fbcc479032b not found: ID does not exist" Nov 04 00:23:47.579955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.579925 6208 scope.go:122] "RemoveContainer" containerID="3eb4f86e887ae23511d74fd2b3666eddc86fcf3c7f45618ee9de9a605f878654" Nov 04 00:23:47.625506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.625410 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-data\") pod \"8390a996-22ca-4972-90d7-c2c6f104c4cc\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " Nov 04 00:23:47.625661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.625553 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-kube-api-access-vdlrs\") pod \"8390a996-22ca-4972-90d7-c2c6f104c4cc\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " Nov 04 00:23:47.625708 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.625657 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-certs\") pod \"8390a996-22ca-4972-90d7-c2c6f104c4cc\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " Nov 04 00:23:47.625746 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.625706 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-data" pod "8390a996-22ca-4972-90d7-c2c6f104c4cc" (UID: "8390a996-22ca-4972-90d7-c2c6f104c4cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:47.625783 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.625769 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-envoy\") pod \"8390a996-22ca-4972-90d7-c2c6f104c4cc\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " Nov 04 00:23:47.625855 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.625841 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-podinfo\") pod \"8390a996-22ca-4972-90d7-c2c6f104c4cc\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " Nov 04 00:23:47.625911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.625871 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-credential-socket\") pod \"8390a996-22ca-4972-90d7-c2c6f104c4cc\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " Nov 04 00:23:47.625972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.625914 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-socket\") pod \"8390a996-22ca-4972-90d7-c2c6f104c4cc\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " Nov 04 00:23:47.625972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.625927 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-certs" pod "8390a996-22ca-4972-90d7-c2c6f104c4cc" (UID: "8390a996-22ca-4972-90d7-c2c6f104c4cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:47.626133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.626101 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8390a996-22ca-4972-90d7-c2c6f104c4cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390a996-22ca-4972-90d7-c2c6f104c4cc-istiod-ca-cert\") pod \"8390a996-22ca-4972-90d7-c2c6f104c4cc\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " Nov 04 00:23:47.626201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.626150 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-token\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-token\") pod \"8390a996-22ca-4972-90d7-c2c6f104c4cc\" (UID: \"8390a996-22ca-4972-90d7-c2c6f104c4cc\") " Nov 04 00:23:47.626272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.626196 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-socket" pod "8390a996-22ca-4972-90d7-c2c6f104c4cc" (UID: "8390a996-22ca-4972-90d7-c2c6f104c4cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:47.626272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.626211 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-credential-socket" pod "8390a996-22ca-4972-90d7-c2c6f104c4cc" (UID: "8390a996-22ca-4972-90d7-c2c6f104c4cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:47.626512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.626474 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8390a996-22ca-4972-90d7-c2c6f104c4cc-istiod-ca-cert" pod "8390a996-22ca-4972-90d7-c2c6f104c4cc" (UID: "8390a996-22ca-4972-90d7-c2c6f104c4cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:23:47.627061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.627010 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:47.627061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.627057 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:47.627246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.627073 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390a996-22ca-4972-90d7-c2c6f104c4cc-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:47.627246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.627087 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:47.627246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.627100 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:47.628263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.628238 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-podinfo" pod "8390a996-22ca-4972-90d7-c2c6f104c4cc" (UID: "8390a996-22ca-4972-90d7-c2c6f104c4cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:23:47.628551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.628520 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-envoy" pod "8390a996-22ca-4972-90d7-c2c6f104c4cc" (UID: "8390a996-22ca-4972-90d7-c2c6f104c4cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:47.628551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.628531 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-kube-api-access-vdlrs" pod "8390a996-22ca-4972-90d7-c2c6f104c4cc" (UID: "8390a996-22ca-4972-90d7-c2c6f104c4cc"). InnerVolumeSpecName "kube-api-access-vdlrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:47.628686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.628562 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-token" pod "8390a996-22ca-4972-90d7-c2c6f104c4cc" (UID: "8390a996-22ca-4972-90d7-c2c6f104c4cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:47.728505 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.728471 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:47.728644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.728508 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdlrs\" (UniqueName: \"kubernetes.io/projected/8390a996-22ca-4972-90d7-c2c6f104c4cc-kube-api-access-vdlrs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:47.728644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.728523 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:47.728644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.728539 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390a996-22ca-4972-90d7-c2c6f104c4cc-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:47.822271 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.822223 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:47.958605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.958561 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh"] Nov 04 00:23:47.965090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.964731 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:23:47.966082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:47.966053 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-2m5qh"] Nov 04 00:23:48.147970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.147932 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-kube-api-access-5l7k7\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-kube-api-access-5l7k7\") pod \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " Nov 04 00:23:48.148174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148029 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-credential-socket\") pod \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " Nov 04 00:23:48.148174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148126 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-data\") pod \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " Nov 04 00:23:48.148281 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148212 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-podinfo\") pod \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " Nov 04 00:23:48.148350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148334 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-certs\") pod \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " Nov 04 00:23:48.148472 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148455 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istiod-ca-cert\") pod \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " Nov 04 00:23:48.148544 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148517 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-envoy\") pod \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " Nov 04 00:23:48.148593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148579 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-token\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-token\") pod \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " Nov 04 00:23:48.148633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148612 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-socket\") pod \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\" (UID: \"3fb44f32-c801-4a3b-b97b-81e776ddbbd6\") " Nov 04 00:23:48.148814 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148776 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-credential-socket" pod "3fb44f32-c801-4a3b-b97b-81e776ddbbd6" (UID: "3fb44f32-c801-4a3b-b97b-81e776ddbbd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:48.148962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.148874 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-data" pod "3fb44f32-c801-4a3b-b97b-81e776ddbbd6" (UID: "3fb44f32-c801-4a3b-b97b-81e776ddbbd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:48.149162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.149136 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-socket" pod "3fb44f32-c801-4a3b-b97b-81e776ddbbd6" (UID: "3fb44f32-c801-4a3b-b97b-81e776ddbbd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:48.149301 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.149199 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-certs" pod "3fb44f32-c801-4a3b-b97b-81e776ddbbd6" (UID: "3fb44f32-c801-4a3b-b97b-81e776ddbbd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:48.149542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.149515 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:48.149650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.149544 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:48.149650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.149573 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:48.149650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.149580 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istiod-ca-cert" pod "3fb44f32-c801-4a3b-b97b-81e776ddbbd6" (UID: "3fb44f32-c801-4a3b-b97b-81e776ddbbd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:23:48.149650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.149586 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:48.150789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.150767 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-kube-api-access-5l7k7" pod "3fb44f32-c801-4a3b-b97b-81e776ddbbd6" (UID: "3fb44f32-c801-4a3b-b97b-81e776ddbbd6"). InnerVolumeSpecName "kube-api-access-5l7k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:48.151012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.150993 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-token" pod "3fb44f32-c801-4a3b-b97b-81e776ddbbd6" (UID: "3fb44f32-c801-4a3b-b97b-81e776ddbbd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:23:48.151147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.151125 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-podinfo" pod "3fb44f32-c801-4a3b-b97b-81e776ddbbd6" (UID: "3fb44f32-c801-4a3b-b97b-81e776ddbbd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:23:48.151322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.151300 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-envoy" pod "3fb44f32-c801-4a3b-b97b-81e776ddbbd6" (UID: "3fb44f32-c801-4a3b-b97b-81e776ddbbd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:23:48.250932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.250902 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:48.250932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.250929 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:48.251139 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.250942 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l7k7\" (UniqueName: \"kubernetes.io/projected/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-kube-api-access-5l7k7\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:48.251139 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.250952 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:48.251139 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.250961 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb44f32-c801-4a3b-b97b-81e776ddbbd6-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:23:48.579154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.579122 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" Nov 04 00:23:48.579621 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.579153 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq" event={"ID":"3fb44f32-c801-4a3b-b97b-81e776ddbbd6","Type":"ContainerDied","Data":"760ea9bd4b504487fcf58fd5209a631e48883530064ee69df28e43e6d6629027"} Nov 04 00:23:48.579621 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.579209 6208 scope.go:122] "RemoveContainer" containerID="c5465e3d6ea30942d305d7f2260dac0b8fa21d82f1f06676fb049102ca0a6100" Nov 04 00:23:48.606047 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.606003 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq"] Nov 04 00:23:48.610303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.610275 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-7v8dq"] Nov 04 00:23:48.672197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:48.672163 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:49.297902 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:49.297867 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" path="/var/lib/kubelet/pods/3fb44f32-c801-4a3b-b97b-81e776ddbbd6/volumes" Nov 04 00:23:49.298458 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:49.298441 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" path="/var/lib/kubelet/pods/8390a996-22ca-4972-90d7-c2c6f104c4cc/volumes" Nov 04 00:23:49.445118 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:49.445084 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:50.154696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:50.154659 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:50.930861 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:50.930822 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:51.653717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:51.653681 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:52.421486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:52.421452 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:53.128011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:53.127971 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:53.805701 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:53.805669 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:53.907484 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:53.907453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:54.599067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:54.599021 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:55.347203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:55.347168 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:56.231996 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:56.231949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:57.035380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:57.035347 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:57.828514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:57.828482 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:58.758227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:58.758180 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:23:59.744322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:23:59.744282 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:00.543423 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:00.543383 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:01.331810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:01.331774 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:02.024866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:02.024826 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:02.921445 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:02.921408 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:03.637619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:03.637587 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:04.345475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:04.345440 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:05.145179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:05.145147 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:05.709844 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:24:05.709807 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:24:06.094699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:06.094606 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:06.822628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:06.822590 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:07.544296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:07.544262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:08.261388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:08.261356 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:09.109943 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:09.109906 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:09.926570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:09.926528 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:10.630963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:10.630929 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:11.430532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:11.430494 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:12.136629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:12.136585 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:12.924515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:12.924480 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:13.744012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:13.743967 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:14.452927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:14.452899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:15.250610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:15.250572 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:15.545187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:15.545091 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:15.645268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:15.645228 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:15.965173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:15.965142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:16.692493 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:16.692462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:17.523076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:17.518477 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:18.224469 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:18.224433 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:18.965250 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:18.965212 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:19.839532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:19.839491 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:20.553624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:20.553590 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:21.341900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:21.341863 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:22.048773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:22.048719 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:22.917098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:22.917062 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:23.745681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:23.745617 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:24.513452 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:24.513415 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:25.331742 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:25.331703 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:26.050234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:26.050199 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:27.021055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:27.017649 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:27.729171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:27.729119 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:28.476584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:28.476497 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:29.223740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:29.223710 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:29.937459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:29.937418 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:30.642634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:30.642594 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:31.428717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:31.428680 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:32.144496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:32.144458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:33.123877 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:33.123831 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:33.692817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:33.692753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:34.042425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:34.042388 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:34.884671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:34.884627 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:35.709251 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:24:35.709213 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:24:35.751441 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:35.751412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:36.493619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:36.493584 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:37.437687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:37.437645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:38.211496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:38.211465 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:38.764027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:38.763986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:39.421054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:39.414650 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:40.136248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:40.136211 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:40.944942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:40.944907 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:41.970177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:41.970126 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:42.281568 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:42.281468 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:42.386500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:42.386448 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:42.906867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:42.906831 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:24:42.907437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:42.907405 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:24:42.914296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:42.914264 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:43.218480 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:43.218447 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:43.734455 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:43.734421 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:44.661556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:44.661517 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:45.691124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:45.691077 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:46.775260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:46.775162 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:47.250912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:47.250575 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:47.851151 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:47.849857 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:48.202257 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:48.202139 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:24:48.202557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:48.202507 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:24:48.669059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:48.668613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:49.139527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:49.139490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:49.789443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:49.789406 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:50.732006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:50.731972 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:51.462728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:51.462694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:52.306722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:52.306684 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:53.183222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:53.183174 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:53.864345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:53.864311 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:53.964518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:53.964480 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:54.823150 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:54.823114 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:55.638992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:55.638950 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:55.743983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:55.743944 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:56.131987 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:56.131956 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:56.359766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:56.359730 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:57.238187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:57.238151 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:58.036650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:58.036616 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:58.945529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:58.945489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:59.579224 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:59.579187 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:24:59.759257 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:24:59.759220 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:00.635191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:00.635155 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:01.440182 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:01.440148 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:02.244558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:02.244522 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:02.953110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:02.953074 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:03.836498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:03.836447 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:03.939069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:03.939026 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:04.667460 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:04.667423 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:05.058255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:05.058219 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:05.154389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:05.154339 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:05.717953 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:25:05.717910 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:25:05.824666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:05.824627 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:06.564813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:06.564780 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:06.667211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:06.667176 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:07.357708 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:07.357678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:08.143633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:08.143600 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:08.979203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:08.979166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:09.724447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:09.724413 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:10.473612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:10.473579 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:10.628972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:10.628938 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:11.525889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:11.525850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:12.183537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:12.183500 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s"] Nov 04 00:25:12.183755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:12.183732 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" containerID="cri-o://7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04" gracePeriod=30 Nov 04 00:25:12.254672 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:12.254621 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:13.042313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:13.042283 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:13.722824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:13.722784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:13.823676 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:13.823629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:14.566107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:14.566070 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:15.379011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:15.378980 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:16.237884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:16.237852 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:17.039023 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.038978 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:17.481221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.481193 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 04 00:25:17.624143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.622925 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-envoy\") pod \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " Nov 04 00:25:17.624143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.622992 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-socket\") pod \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " Nov 04 00:25:17.624143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.623017 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-certs\") pod \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " Nov 04 00:25:17.624143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.623151 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istiod-ca-cert\") pod \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " Nov 04 00:25:17.624143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.623252 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-data\") pod \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " Nov 04 00:25:17.624143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.623387 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-podinfo\") pod \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " Nov 04 00:25:17.624143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.623496 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-token\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-token\") pod \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " Nov 04 00:25:17.624143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.623639 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-credential-socket\") pod \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " Nov 04 00:25:17.624143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.623720 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-kube-api-access-2nt7f\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-kube-api-access-2nt7f\") pod \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\" (UID: \"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0\") " Nov 04 00:25:17.627548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.627009 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istiod-ca-cert" pod "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" (UID: "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:25:17.627548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.627349 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-certs" pod "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" (UID: "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:17.628587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.628362 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-data" pod "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" (UID: "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:17.628587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.628560 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-credential-socket" pod "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" (UID: "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:17.628751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.628577 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-socket" pod "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" (UID: "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:17.630720 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.630695 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-kube-api-access-2nt7f" pod "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" (UID: "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0"). InnerVolumeSpecName "kube-api-access-2nt7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:25:17.631205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.631180 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-envoy" pod "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" (UID: "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:17.631791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.631772 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-podinfo" pod "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" (UID: "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:25:17.631996 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.631982 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-token" pod "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" (UID: "9e7485bb-2460-4c40-8815-bcfaf0a2c9a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:25:17.728299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.728253 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:17.728299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.728292 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:17.728524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.728308 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:17.728524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.728323 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nt7f\" (UniqueName: \"kubernetes.io/projected/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-kube-api-access-2nt7f\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:17.728524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.728337 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:17.728524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.728351 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:17.728524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.728365 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:17.728524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.728379 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:17.728524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.728394 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:17.946907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.946820 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:17.970228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.970192 6208 generic.go:356] "Generic (PLEG): container finished" podID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerID="7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04" exitCode=0 Nov 04 00:25:17.970408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.970289 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" Nov 04 00:25:17.970408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.970311 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" event={"ID":"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0","Type":"ContainerDied","Data":"7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04"} Nov 04 00:25:17.970408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.970377 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s" event={"ID":"9e7485bb-2460-4c40-8815-bcfaf0a2c9a0","Type":"ContainerDied","Data":"595771f70c81b685826f95e7ddde95b4ae34d15be334d4e01892d0d5dade9d72"} Nov 04 00:25:17.970531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.970412 6208 scope.go:122] "RemoveContainer" containerID="7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04" Nov 04 00:25:17.985520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:17.985493 6208 scope.go:122] "RemoveContainer" containerID="30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081" Nov 04 00:25:18.041904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:18.040653 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s"] Nov 04 00:25:18.053803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:18.053762 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2341ca-openshift-default-79655985d9-8kw7s"] Nov 04 00:25:18.073987 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:18.073957 6208 scope.go:122] "RemoveContainer" containerID="7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04" Nov 04 00:25:18.075090 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:25:18.074390 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04\": container with ID starting with 7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04 not found: ID does not exist" containerID="7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04" Nov 04 00:25:18.075090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:18.074424 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04"} err="failed to get container status \"7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04\": rpc error: code = NotFound desc = could not find container \"7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04\": container with ID starting with 7265c0d74e34282c42fdae41e1844f888da50e409617dc9ee7dcc704ce261c04 not found: ID does not exist" Nov 04 00:25:18.075090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:18.074442 6208 scope.go:122] "RemoveContainer" containerID="30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081" Nov 04 00:25:18.075090 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:25:18.074714 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081\": container with ID starting with 30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081 not found: ID does not exist" containerID="30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081" Nov 04 00:25:18.075090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:18.074738 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081"} err="failed to get container status \"30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081\": rpc error: code = NotFound desc = could not find container \"30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081\": container with ID starting with 30b66158fd027be6303534d7aba256ff8b6cc58cddd2fa5f58a9f60c48865081 not found: ID does not exist" Nov 04 00:25:18.557831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:18.557788 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6"] Nov 04 00:25:18.558376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:18.558073 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" podUID="46d8c34f-fd93-4700-8430-f75fcc6a670f" containerName="maas-api" containerID="cri-o://5fbfcdf72d82ba7dd7de926a985b191059d3eaaaaf022bbdead056d8f48f7b63" gracePeriod=30 Nov 04 00:25:18.605234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:18.605188 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:19.057048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.056620 6208 generic.go:356] "Generic (PLEG): container finished" podID="46d8c34f-fd93-4700-8430-f75fcc6a670f" containerID="5fbfcdf72d82ba7dd7de926a985b191059d3eaaaaf022bbdead056d8f48f7b63" exitCode=0 Nov 04 00:25:19.057048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.056824 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" event={"ID":"46d8c34f-fd93-4700-8430-f75fcc6a670f","Type":"ContainerDied","Data":"5fbfcdf72d82ba7dd7de926a985b191059d3eaaaaf022bbdead056d8f48f7b63"} Nov 04 00:25:19.223193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.223167 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:25:19.235216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.235189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:19.273658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.273621 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d8c34f-fd93-4700-8430-f75fcc6a670f-kube-api-access-rjxfh\" (UniqueName: \"kubernetes.io/projected/46d8c34f-fd93-4700-8430-f75fcc6a670f-kube-api-access-rjxfh\") pod \"46d8c34f-fd93-4700-8430-f75fcc6a670f\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " Nov 04 00:25:19.273840 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.273682 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-maas-api-tls\") pod \"46d8c34f-fd93-4700-8430-f75fcc6a670f\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " Nov 04 00:25:19.273881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.273857 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-metrics-certs\") pod \"46d8c34f-fd93-4700-8430-f75fcc6a670f\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " Nov 04 00:25:19.274084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.274027 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d8c34f-fd93-4700-8430-f75fcc6a670f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d8c34f-fd93-4700-8430-f75fcc6a670f-openshift-service-ca\") pod \"46d8c34f-fd93-4700-8430-f75fcc6a670f\" (UID: \"46d8c34f-fd93-4700-8430-f75fcc6a670f\") " Nov 04 00:25:19.274413 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.274384 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d8c34f-fd93-4700-8430-f75fcc6a670f-openshift-service-ca" pod "46d8c34f-fd93-4700-8430-f75fcc6a670f" (UID: "46d8c34f-fd93-4700-8430-f75fcc6a670f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:25:19.274891 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.274863 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d8c34f-fd93-4700-8430-f75fcc6a670f-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:19.276060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.276002 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d8c34f-fd93-4700-8430-f75fcc6a670f-kube-api-access-rjxfh" pod "46d8c34f-fd93-4700-8430-f75fcc6a670f" (UID: "46d8c34f-fd93-4700-8430-f75fcc6a670f"). InnerVolumeSpecName "kube-api-access-rjxfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:25:19.276160 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.276058 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-metrics-certs" pod "46d8c34f-fd93-4700-8430-f75fcc6a670f" (UID: "46d8c34f-fd93-4700-8430-f75fcc6a670f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:25:19.276160 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.276105 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-maas-api-tls" pod "46d8c34f-fd93-4700-8430-f75fcc6a670f" (UID: "46d8c34f-fd93-4700-8430-f75fcc6a670f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:25:19.298025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.297993 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" path="/var/lib/kubelet/pods/9e7485bb-2460-4c40-8815-bcfaf0a2c9a0/volumes" Nov 04 00:25:19.380235 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.378407 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjxfh\" (UniqueName: \"kubernetes.io/projected/46d8c34f-fd93-4700-8430-f75fcc6a670f-kube-api-access-rjxfh\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:19.380235 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.378458 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:19.380235 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.378472 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d8c34f-fd93-4700-8430-f75fcc6a670f-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:19.952080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:19.952016 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:20.066843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:20.066813 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" Nov 04 00:25:20.066843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:20.066828 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6" event={"ID":"46d8c34f-fd93-4700-8430-f75fcc6a670f","Type":"ContainerDied","Data":"e3e09957c21c659de31c8285cb2a7e36e19ad9e27e5dfe9e0ef53d00b7714339"} Nov 04 00:25:20.067951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:20.066915 6208 scope.go:122] "RemoveContainer" containerID="5fbfcdf72d82ba7dd7de926a985b191059d3eaaaaf022bbdead056d8f48f7b63" Nov 04 00:25:20.087549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:20.087516 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6"] Nov 04 00:25:20.091456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:20.091431 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-954cd9-6fbb47d76f-xkft6"] Nov 04 00:25:20.595190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:20.595152 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:20.711055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:20.708795 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:20.932129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:20.928682 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:21.196872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:21.196784 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:21.297762 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:21.297730 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d8c34f-fd93-4700-8430-f75fcc6a670f" path="/var/lib/kubelet/pods/46d8c34f-fd93-4700-8430-f75fcc6a670f/volumes" Nov 04 00:25:21.542500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:21.542461 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92"] Nov 04 00:25:21.542787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:21.542753 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" containerID="cri-o://b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846" gracePeriod=30 Nov 04 00:25:21.571217 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:21.571184 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:22.176932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:22.176891 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:22.386162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:22.386127 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:22.942096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:22.942024 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:23.573816 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:23.573772 6208 patch_prober.go:29] interesting pod/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:25:23.574258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:23.573837 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:25:23.652381 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:23.652341 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:24.444896 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:24.444835 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:25.347743 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:25.347704 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:25.959280 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:25.959245 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:26.843637 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.843598 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:26.848476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.848453 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:25:26.981185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981140 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-credential-socket\") pod \"555966c0-f453-4441-8991-cf25c1200236\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " Nov 04 00:25:26.981361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981231 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-istio-token\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-istio-token\") pod \"555966c0-f453-4441-8991-cf25c1200236\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " Nov 04 00:25:26.981361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981293 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-socket\") pod \"555966c0-f453-4441-8991-cf25c1200236\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " Nov 04 00:25:26.981361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981324 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-kube-api-access-d2mg2\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-kube-api-access-d2mg2\") pod \"555966c0-f453-4441-8991-cf25c1200236\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " Nov 04 00:25:26.981514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981387 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-envoy\") pod \"555966c0-f453-4441-8991-cf25c1200236\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " Nov 04 00:25:26.981514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981445 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-data\") pod \"555966c0-f453-4441-8991-cf25c1200236\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " Nov 04 00:25:26.981622 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981514 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-credential-socket" pod "555966c0-f453-4441-8991-cf25c1200236" (UID: "555966c0-f453-4441-8991-cf25c1200236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:26.981691 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981627 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-socket" pod "555966c0-f453-4441-8991-cf25c1200236" (UID: "555966c0-f453-4441-8991-cf25c1200236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:26.981771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981748 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/555966c0-f453-4441-8991-cf25c1200236-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555966c0-f453-4441-8991-cf25c1200236-istio-podinfo\") pod \"555966c0-f453-4441-8991-cf25c1200236\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " Nov 04 00:25:26.981836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981756 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-data" pod "555966c0-f453-4441-8991-cf25c1200236" (UID: "555966c0-f453-4441-8991-cf25c1200236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:26.981836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981794 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555966c0-f453-4441-8991-cf25c1200236-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555966c0-f453-4441-8991-cf25c1200236-istiod-ca-cert\") pod \"555966c0-f453-4441-8991-cf25c1200236\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " Nov 04 00:25:26.981975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.981954 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-certs\") pod \"555966c0-f453-4441-8991-cf25c1200236\" (UID: \"555966c0-f453-4441-8991-cf25c1200236\") " Nov 04 00:25:26.982129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.982099 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555966c0-f453-4441-8991-cf25c1200236-istiod-ca-cert" pod "555966c0-f453-4441-8991-cf25c1200236" (UID: "555966c0-f453-4441-8991-cf25c1200236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:25:26.982286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.982242 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-certs" pod "555966c0-f453-4441-8991-cf25c1200236" (UID: "555966c0-f453-4441-8991-cf25c1200236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:26.983819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.983797 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-istio-token" pod "555966c0-f453-4441-8991-cf25c1200236" (UID: "555966c0-f453-4441-8991-cf25c1200236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:25:26.984163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984128 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-kube-api-access-d2mg2" pod "555966c0-f453-4441-8991-cf25c1200236" (UID: "555966c0-f453-4441-8991-cf25c1200236"). InnerVolumeSpecName "kube-api-access-d2mg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:25:26.984328 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984301 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/555966c0-f453-4441-8991-cf25c1200236-istio-podinfo" pod "555966c0-f453-4441-8991-cf25c1200236" (UID: "555966c0-f453-4441-8991-cf25c1200236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:25:26.984466 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984446 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555966c0-f453-4441-8991-cf25c1200236-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:26.984539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984460 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-envoy" pod "555966c0-f453-4441-8991-cf25c1200236" (UID: "555966c0-f453-4441-8991-cf25c1200236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:26.984539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984472 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555966c0-f453-4441-8991-cf25c1200236-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:26.984539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984517 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:26.984539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984533 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:26.984776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984548 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:26.984776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984562 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:26.984776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984577 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2mg2\" (UniqueName: \"kubernetes.io/projected/555966c0-f453-4441-8991-cf25c1200236-kube-api-access-d2mg2\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:26.984776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:26.984592 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:27.085949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.085862 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555966c0-f453-4441-8991-cf25c1200236-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:27.142884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.142850 6208 generic.go:356] "Generic (PLEG): container finished" podID="555966c0-f453-4441-8991-cf25c1200236" containerID="b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846" exitCode=0 Nov 04 00:25:27.143113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.142912 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" Nov 04 00:25:27.143113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.142942 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" event={"ID":"555966c0-f453-4441-8991-cf25c1200236","Type":"ContainerDied","Data":"b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846"} Nov 04 00:25:27.143113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.143002 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92" event={"ID":"555966c0-f453-4441-8991-cf25c1200236","Type":"ContainerDied","Data":"d16a192095da4c8b5cc0bb61d59ccfbaf03a506b130a82a15ee0e06211e911dd"} Nov 04 00:25:27.143113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.143024 6208 scope.go:122] "RemoveContainer" containerID="b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846" Nov 04 00:25:27.156576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.156554 6208 scope.go:122] "RemoveContainer" containerID="cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a" Nov 04 00:25:27.169797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.169610 6208 scope.go:122] "RemoveContainer" containerID="b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846" Nov 04 00:25:27.170085 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:25:27.170021 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846\": container with ID starting with b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846 not found: ID does not exist" containerID="b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846" Nov 04 00:25:27.170211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.170093 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846"} err="failed to get container status \"b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846\": rpc error: code = NotFound desc = could not find container \"b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846\": container with ID starting with b4a0951d7d2f71d7c0840eba80fc2c7fe21c491196bd3edf911e9f0ebe3d4846 not found: ID does not exist" Nov 04 00:25:27.170211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.170125 6208 scope.go:122] "RemoveContainer" containerID="cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a" Nov 04 00:25:27.170440 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:25:27.170419 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a\": container with ID starting with cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a not found: ID does not exist" containerID="cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a" Nov 04 00:25:27.170484 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.170446 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a"} err="failed to get container status \"cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a\": rpc error: code = NotFound desc = could not find container \"cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a\": container with ID starting with cabcc5ce912dfdc68001657290cd3d527825859a98789bced4aa6c11734aa44a not found: ID does not exist" Nov 04 00:25:27.171389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.171025 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92"] Nov 04 00:25:27.175283 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.175262 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-954cd9-openshift-default-5f7d76cdf5-9kw92"] Nov 04 00:25:27.296073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.296023 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555966c0-f453-4441-8991-cf25c1200236" path="/var/lib/kubelet/pods/555966c0-f453-4441-8991-cf25c1200236/volumes" Nov 04 00:25:27.620583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:27.620542 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:28.095053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:28.094572 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:28.193935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:28.193901 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:28.741967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:28.741929 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:29.347391 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:29.347361 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:29.521069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:29.520999 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:30.133791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:30.133753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:30.860116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:30.860022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:31.648958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:31.648920 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:32.354529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:32.354499 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:33.058929 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:33.058894 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:33.163091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:33.162994 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:33.856222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:33.856183 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:34.615860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:34.615826 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:35.647698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:35.647663 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:35.721662 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:25:35.721624 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:25:36.536825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:36.536783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:37.298264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:37.298227 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:37.423674 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:37.423599 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:38.112843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:38.112804 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:38.226055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:38.222186 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:38.618502 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:38.618460 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:39.324328 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:39.324296 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:40.059185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:40.059151 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:40.824335 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:40.824302 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:41.628288 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:41.628249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:42.347643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:42.347613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:43.053085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:43.053022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:43.271218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:43.271183 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:43.933958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:43.933920 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:44.631753 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:44.631714 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:45.353754 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:45.353721 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:46.112096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:46.112055 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:46.831536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:46.831491 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:47.289048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:47.289004 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:47.489051 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:47.489009 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt"] Nov 04 00:25:47.489313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:47.489285 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" podUID="28b0b70f-ad5e-47e5-af92-f743b51805ad" containerName="maas-api" containerID="cri-o://8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b" gracePeriod=30 Nov 04 00:25:47.531875 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:47.531845 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:48.053116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.053091 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:25:48.090885 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.090850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:48.126998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.126969 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-maas-api-tls\") pod \"28b0b70f-ad5e-47e5-af92-f743b51805ad\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " Nov 04 00:25:48.127171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.127122 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-metrics-certs\") pod \"28b0b70f-ad5e-47e5-af92-f743b51805ad\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " Nov 04 00:25:48.127296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.127281 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b0b70f-ad5e-47e5-af92-f743b51805ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0b70f-ad5e-47e5-af92-f743b51805ad-openshift-service-ca\") pod \"28b0b70f-ad5e-47e5-af92-f743b51805ad\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " Nov 04 00:25:48.127443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.127396 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b0b70f-ad5e-47e5-af92-f743b51805ad-kube-api-access-pclnk\" (UniqueName: \"kubernetes.io/projected/28b0b70f-ad5e-47e5-af92-f743b51805ad-kube-api-access-pclnk\") pod \"28b0b70f-ad5e-47e5-af92-f743b51805ad\" (UID: \"28b0b70f-ad5e-47e5-af92-f743b51805ad\") " Nov 04 00:25:48.127642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.127617 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b0b70f-ad5e-47e5-af92-f743b51805ad-openshift-service-ca" pod "28b0b70f-ad5e-47e5-af92-f743b51805ad" (UID: "28b0b70f-ad5e-47e5-af92-f743b51805ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:25:48.128232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.128203 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28b0b70f-ad5e-47e5-af92-f743b51805ad-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:48.129114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.129091 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-maas-api-tls" pod "28b0b70f-ad5e-47e5-af92-f743b51805ad" (UID: "28b0b70f-ad5e-47e5-af92-f743b51805ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:25:48.129403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.129387 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-metrics-certs" pod "28b0b70f-ad5e-47e5-af92-f743b51805ad" (UID: "28b0b70f-ad5e-47e5-af92-f743b51805ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:25:48.129593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.129571 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b0b70f-ad5e-47e5-af92-f743b51805ad-kube-api-access-pclnk" pod "28b0b70f-ad5e-47e5-af92-f743b51805ad" (UID: "28b0b70f-ad5e-47e5-af92-f743b51805ad"). InnerVolumeSpecName "kube-api-access-pclnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:25:48.233263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.230471 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:48.233263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.230514 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pclnk\" (UniqueName: \"kubernetes.io/projected/28b0b70f-ad5e-47e5-af92-f743b51805ad-kube-api-access-pclnk\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:48.233263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.230529 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28b0b70f-ad5e-47e5-af92-f743b51805ad-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:48.351525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.351441 6208 generic.go:356] "Generic (PLEG): container finished" podID="28b0b70f-ad5e-47e5-af92-f743b51805ad" containerID="8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b" exitCode=0 Nov 04 00:25:48.351525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.351484 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" Nov 04 00:25:48.351994 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.351523 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" event={"ID":"28b0b70f-ad5e-47e5-af92-f743b51805ad","Type":"ContainerDied","Data":"8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b"} Nov 04 00:25:48.351994 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.351573 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt" event={"ID":"28b0b70f-ad5e-47e5-af92-f743b51805ad","Type":"ContainerDied","Data":"83de62c7dbda8403e0a3f928fdc4fbceccf43b81e1557884361daf2e20782be6"} Nov 04 00:25:48.351994 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.351597 6208 scope.go:122] "RemoveContainer" containerID="8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b" Nov 04 00:25:48.366239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.366220 6208 scope.go:122] "RemoveContainer" containerID="8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b" Nov 04 00:25:48.366516 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:25:48.366497 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b\": container with ID starting with 8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b not found: ID does not exist" containerID="8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b" Nov 04 00:25:48.366655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.366526 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b"} err="failed to get container status \"8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b\": rpc error: code = NotFound desc = could not find container \"8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b\": container with ID starting with 8cb143d66cbdf4accdf52e3b51a910f9a34cfd21cf49614f95af00abf563ea7b not found: ID does not exist" Nov 04 00:25:48.379150 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.379128 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt"] Nov 04 00:25:48.382938 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.382918 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a7cbc3-556988587c-fb7pt"] Nov 04 00:25:48.824901 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.824862 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:48.983017 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.982982 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74"] Nov 04 00:25:48.983255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:48.983222 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" containerID="cri-o://f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8" gracePeriod=30 Nov 04 00:25:49.010972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:49.010942 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:49.323220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:49.322477 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b0b70f-ad5e-47e5-af92-f743b51805ad" path="/var/lib/kubelet/pods/28b0b70f-ad5e-47e5-af92-f743b51805ad/volumes" Nov 04 00:25:49.716026 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:49.715997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:50.446092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:50.446051 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:50.596071 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:50.596002 6208 patch_prober.go:29] interesting pod/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:25:50.596242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:50.596084 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:25:51.406631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:51.406595 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:52.147358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:52.147320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:52.916864 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:52.916832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:53.637899 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:53.637863 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:54.187048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.186997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:54.288473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.288347 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:54.325153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.325126 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:25:54.393966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.393927 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:54.411664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.411636 6208 generic.go:356] "Generic (PLEG): container finished" podID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerID="f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8" exitCode=0 Nov 04 00:25:54.411844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.411696 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" Nov 04 00:25:54.411844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.411726 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" event={"ID":"e8e32687-55f2-4eb8-bf9d-af860aa9848e","Type":"ContainerDied","Data":"f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8"} Nov 04 00:25:54.411844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.411799 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74" event={"ID":"e8e32687-55f2-4eb8-bf9d-af860aa9848e","Type":"ContainerDied","Data":"b5b99398e864262ac7f6fe8e493850e8cd1c0b047c2d043078bfa1c69523d8cc"} Nov 04 00:25:54.411844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.411832 6208 scope.go:122] "RemoveContainer" containerID="f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8" Nov 04 00:25:54.423674 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.423654 6208 scope.go:122] "RemoveContainer" containerID="f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8" Nov 04 00:25:54.423964 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:25:54.423938 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8\": container with ID starting with f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8 not found: ID does not exist" containerID="f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8" Nov 04 00:25:54.424016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.423967 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8"} err="failed to get container status \"f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8\": rpc error: code = NotFound desc = could not find container \"f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8\": container with ID starting with f635d8a648be4605422399bfd6e82a0daa7bf578bf2b82c6d60e84a44e724eb8 not found: ID does not exist" Nov 04 00:25:54.500912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.500872 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-kube-api-access-ksffc\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-kube-api-access-ksffc\") pod \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " Nov 04 00:25:54.501182 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.500931 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-envoy\") pod \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " Nov 04 00:25:54.501261 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.501182 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-credential-socket\") pod \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " Nov 04 00:25:54.501261 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.501247 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-socket\") pod \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " Nov 04 00:25:54.501367 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.501302 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-certs\") pod \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " Nov 04 00:25:54.501367 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.501364 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-token\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-token\") pod \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " Nov 04 00:25:54.501473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.501383 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-data\") pod \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " Nov 04 00:25:54.501473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.501402 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istiod-ca-cert\") pod \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " Nov 04 00:25:54.501582 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.501501 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-podinfo\") pod \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\" (UID: \"e8e32687-55f2-4eb8-bf9d-af860aa9848e\") " Nov 04 00:25:54.502944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.502901 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-socket" pod "e8e32687-55f2-4eb8-bf9d-af860aa9848e" (UID: "e8e32687-55f2-4eb8-bf9d-af860aa9848e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:54.503134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.502958 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-credential-socket" pod "e8e32687-55f2-4eb8-bf9d-af860aa9848e" (UID: "e8e32687-55f2-4eb8-bf9d-af860aa9848e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:54.503228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.503209 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-data" pod "e8e32687-55f2-4eb8-bf9d-af860aa9848e" (UID: "e8e32687-55f2-4eb8-bf9d-af860aa9848e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:54.503427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.503397 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-certs" pod "e8e32687-55f2-4eb8-bf9d-af860aa9848e" (UID: "e8e32687-55f2-4eb8-bf9d-af860aa9848e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:54.503520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.503491 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istiod-ca-cert" pod "e8e32687-55f2-4eb8-bf9d-af860aa9848e" (UID: "e8e32687-55f2-4eb8-bf9d-af860aa9848e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:25:54.505169 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.505138 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-token" pod "e8e32687-55f2-4eb8-bf9d-af860aa9848e" (UID: "e8e32687-55f2-4eb8-bf9d-af860aa9848e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:25:54.505643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.505619 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-podinfo" pod "e8e32687-55f2-4eb8-bf9d-af860aa9848e" (UID: "e8e32687-55f2-4eb8-bf9d-af860aa9848e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:25:54.505727 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.505680 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-kube-api-access-ksffc" pod "e8e32687-55f2-4eb8-bf9d-af860aa9848e" (UID: "e8e32687-55f2-4eb8-bf9d-af860aa9848e"). InnerVolumeSpecName "kube-api-access-ksffc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:25:54.505812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.505794 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-envoy" pod "e8e32687-55f2-4eb8-bf9d-af860aa9848e" (UID: "e8e32687-55f2-4eb8-bf9d-af860aa9848e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:25:54.603340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.603262 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:54.603340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.603293 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:54.603340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.603303 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:54.603340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.603312 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:54.603340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.603321 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:54.603340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.603331 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:54.603340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.603340 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:54.604221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.603350 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksffc\" (UniqueName: \"kubernetes.io/projected/e8e32687-55f2-4eb8-bf9d-af860aa9848e-kube-api-access-ksffc\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:54.604221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.603358 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e32687-55f2-4eb8-bf9d-af860aa9848e-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:25:54.653364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.653329 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:54.773393 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.773353 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74"] Nov 04 00:25:54.777384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:54.777357 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9a6f7a-openshift-default-547bb785c9-n4k74"] Nov 04 00:25:55.030663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:55.030609 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:55.344215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:55.344123 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" path="/var/lib/kubelet/pods/e8e32687-55f2-4eb8-bf9d-af860aa9848e/volumes" Nov 04 00:25:55.532078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:55.532046 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:55.959165 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:55.959131 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:56.060548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:56.060513 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:56.454084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:56.453995 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:57.258944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:57.258912 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:58.037022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:58.036988 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:58.837698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:58.837659 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:59.628893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:59.628853 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:25:59.839735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:25:59.839703 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:00.450465 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:00.450428 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:01.222822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:01.222786 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:01.958208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:01.958172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:02.741306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:02.741266 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:03.445278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:03.445237 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:04.264958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:04.264921 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:05.054324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:05.054266 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:05.152885 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:05.152844 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:05.708092 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:26:05.708019 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:26:05.936979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:05.936920 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:06.543566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:06.543526 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:07.054946 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:07.054913 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:07.570083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:07.570022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:07.876682 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:07.876590 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:08.643992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:08.643952 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:09.408745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:09.408712 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:10.224889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:10.224855 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:10.355166 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:10.355136 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:11.155089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:11.155020 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:11.924249 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:11.924209 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:12.168722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:12.168690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:12.826230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:12.826194 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:13.629239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:13.629202 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:14.420788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:14.420749 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:14.701647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:14.701560 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:14.801953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:14.801917 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:15.320127 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:15.320020 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:15.423385 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:15.423346 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:15.706144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:15.706114 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768"] Nov 04 00:26:15.706945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:15.706911 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" containerID="cri-o://024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2" gracePeriod=30 Nov 04 00:26:15.710919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:15.710657 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z"] Nov 04 00:26:15.710919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:15.710873 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" containerID="cri-o://afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a" gracePeriod=30 Nov 04 00:26:15.711550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:15.711520 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:16.472538 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:16.472499 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:16.761337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:16.761231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:16.859932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:16.859883 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:17.449882 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:17.449829 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:18.224681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:18.224646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:19.014609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:19.014574 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:19.601027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:19.600992 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:19.823358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:19.823323 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:20.532789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:20.532752 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:21.034475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.034451 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:26:21.048293 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.048263 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:26:21.137136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137045 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-data\") pod \"515f512b-c7a2-4d2c-b443-bfe912c77f80\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " Nov 04 00:26:21.137322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137169 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-certs\") pod \"515f512b-c7a2-4d2c-b443-bfe912c77f80\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " Nov 04 00:26:21.137322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137238 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-kube-api-access-k82wp\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-kube-api-access-k82wp\") pod \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " Nov 04 00:26:21.137322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137269 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-podinfo\") pod \"515f512b-c7a2-4d2c-b443-bfe912c77f80\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " Nov 04 00:26:21.137322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137308 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istiod-ca-cert\") pod \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " Nov 04 00:26:21.137573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137370 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-envoy\") pod \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " Nov 04 00:26:21.137573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137398 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-socket\") pod \"515f512b-c7a2-4d2c-b443-bfe912c77f80\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " Nov 04 00:26:21.137573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137460 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-credential-socket\") pod \"515f512b-c7a2-4d2c-b443-bfe912c77f80\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " Nov 04 00:26:21.137573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137489 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-token\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-token\") pod \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " Nov 04 00:26:21.137573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137520 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-kube-api-access-29b4f\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-kube-api-access-29b4f\") pod \"515f512b-c7a2-4d2c-b443-bfe912c77f80\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " Nov 04 00:26:21.137850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137580 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-envoy\") pod \"515f512b-c7a2-4d2c-b443-bfe912c77f80\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " Nov 04 00:26:21.137850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137610 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-socket\") pod \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " Nov 04 00:26:21.137850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137643 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-data\") pod \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " Nov 04 00:26:21.137850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137740 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515f512b-c7a2-4d2c-b443-bfe912c77f80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515f512b-c7a2-4d2c-b443-bfe912c77f80-istiod-ca-cert\") pod \"515f512b-c7a2-4d2c-b443-bfe912c77f80\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " Nov 04 00:26:21.138113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137857 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-certs\") pod \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " Nov 04 00:26:21.138113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.137920 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-credential-socket\") pod \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " Nov 04 00:26:21.138113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.138024 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-podinfo\") pod \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\" (UID: \"a1516c89-1bc2-4251-90f8-f1d6688dfdf6\") " Nov 04 00:26:21.138565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.138139 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-token\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-token\") pod \"515f512b-c7a2-4d2c-b443-bfe912c77f80\" (UID: \"515f512b-c7a2-4d2c-b443-bfe912c77f80\") " Nov 04 00:26:21.139446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.139289 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-data" pod "515f512b-c7a2-4d2c-b443-bfe912c77f80" (UID: "515f512b-c7a2-4d2c-b443-bfe912c77f80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.139577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.139554 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-socket" pod "a1516c89-1bc2-4251-90f8-f1d6688dfdf6" (UID: "a1516c89-1bc2-4251-90f8-f1d6688dfdf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.139666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.139643 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istiod-ca-cert" pod "a1516c89-1bc2-4251-90f8-f1d6688dfdf6" (UID: "a1516c89-1bc2-4251-90f8-f1d6688dfdf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:26:21.139807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.139786 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-credential-socket" pod "515f512b-c7a2-4d2c-b443-bfe912c77f80" (UID: "515f512b-c7a2-4d2c-b443-bfe912c77f80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.140111 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.140084 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-certs" pod "a1516c89-1bc2-4251-90f8-f1d6688dfdf6" (UID: "a1516c89-1bc2-4251-90f8-f1d6688dfdf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.140211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.140125 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-socket" pod "515f512b-c7a2-4d2c-b443-bfe912c77f80" (UID: "515f512b-c7a2-4d2c-b443-bfe912c77f80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.140318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.140297 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-data" pod "a1516c89-1bc2-4251-90f8-f1d6688dfdf6" (UID: "a1516c89-1bc2-4251-90f8-f1d6688dfdf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.140406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.140287 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-certs" pod "515f512b-c7a2-4d2c-b443-bfe912c77f80" (UID: "515f512b-c7a2-4d2c-b443-bfe912c77f80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.140614 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.140592 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515f512b-c7a2-4d2c-b443-bfe912c77f80-istiod-ca-cert" pod "515f512b-c7a2-4d2c-b443-bfe912c77f80" (UID: "515f512b-c7a2-4d2c-b443-bfe912c77f80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:26:21.140836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.140813 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-credential-socket" pod "a1516c89-1bc2-4251-90f8-f1d6688dfdf6" (UID: "a1516c89-1bc2-4251-90f8-f1d6688dfdf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.143519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.143493 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-kube-api-access-29b4f" pod "515f512b-c7a2-4d2c-b443-bfe912c77f80" (UID: "515f512b-c7a2-4d2c-b443-bfe912c77f80"). InnerVolumeSpecName "kube-api-access-29b4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:26:21.144562 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.144494 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-token" pod "515f512b-c7a2-4d2c-b443-bfe912c77f80" (UID: "515f512b-c7a2-4d2c-b443-bfe912c77f80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:26:21.144562 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.144538 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-kube-api-access-k82wp" pod "a1516c89-1bc2-4251-90f8-f1d6688dfdf6" (UID: "a1516c89-1bc2-4251-90f8-f1d6688dfdf6"). InnerVolumeSpecName "kube-api-access-k82wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:26:21.145323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.145297 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-podinfo" pod "a1516c89-1bc2-4251-90f8-f1d6688dfdf6" (UID: "a1516c89-1bc2-4251-90f8-f1d6688dfdf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:26:21.145683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.145647 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-envoy" pod "515f512b-c7a2-4d2c-b443-bfe912c77f80" (UID: "515f512b-c7a2-4d2c-b443-bfe912c77f80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.146512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.146463 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-envoy" pod "a1516c89-1bc2-4251-90f8-f1d6688dfdf6" (UID: "a1516c89-1bc2-4251-90f8-f1d6688dfdf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:21.146653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.146576 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-podinfo" pod "515f512b-c7a2-4d2c-b443-bfe912c77f80" (UID: "515f512b-c7a2-4d2c-b443-bfe912c77f80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:26:21.147860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.147835 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-token" pod "a1516c89-1bc2-4251-90f8-f1d6688dfdf6" (UID: "a1516c89-1bc2-4251-90f8-f1d6688dfdf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:26:21.240740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240692 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/515f512b-c7a2-4d2c-b443-bfe912c77f80-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.240740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240736 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.240740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240747 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.240740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240755 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240765 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240774 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240782 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240791 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k82wp\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-kube-api-access-k82wp\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240800 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240808 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240819 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240833 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240846 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240858 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240872 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29b4f\" (UniqueName: \"kubernetes.io/projected/515f512b-c7a2-4d2c-b443-bfe912c77f80-kube-api-access-29b4f\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240884 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/515f512b-c7a2-4d2c-b443-bfe912c77f80-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240900 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.241054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.240914 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1516c89-1bc2-4251-90f8-f1d6688dfdf6-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:21.432141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.429951 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:21.553065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.542667 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:21.690897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.690813 6208 generic.go:356] "Generic (PLEG): container finished" podID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerID="afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a" exitCode=0 Nov 04 00:26:21.691081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.690935 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" event={"ID":"515f512b-c7a2-4d2c-b443-bfe912c77f80","Type":"ContainerDied","Data":"afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a"} Nov 04 00:26:21.691081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.690988 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" event={"ID":"515f512b-c7a2-4d2c-b443-bfe912c77f80","Type":"ContainerDied","Data":"113d52bfe41f09f11c74a34d55763b686b0f0536720599c0342cde0200bfd9af"} Nov 04 00:26:21.691764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.691091 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z" Nov 04 00:26:21.692502 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.691108 6208 scope.go:122] "RemoveContainer" containerID="afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a" Nov 04 00:26:21.696004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.695481 6208 generic.go:356] "Generic (PLEG): container finished" podID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerID="024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2" exitCode=0 Nov 04 00:26:21.696004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.695759 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" event={"ID":"a1516c89-1bc2-4251-90f8-f1d6688dfdf6","Type":"ContainerDied","Data":"024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2"} Nov 04 00:26:21.696004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.695797 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" event={"ID":"a1516c89-1bc2-4251-90f8-f1d6688dfdf6","Type":"ContainerDied","Data":"565527d3f3dbfee7c47536258a3a38de0b7300b12e46a699eeaf3397d3ba947e"} Nov 04 00:26:21.696004 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.695874 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768" Nov 04 00:26:21.723807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.723770 6208 scope.go:122] "RemoveContainer" containerID="afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a" Nov 04 00:26:21.724711 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:26:21.724681 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a\": container with ID starting with afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a not found: ID does not exist" containerID="afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a" Nov 04 00:26:21.724826 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.724722 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a"} err="failed to get container status \"afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a\": rpc error: code = NotFound desc = could not find container \"afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a\": container with ID starting with afa7c004d8002149c915a8649f4ccf97b967c75fd6ca35308eada73f5f556f0a not found: ID does not exist" Nov 04 00:26:21.724826 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.724743 6208 scope.go:122] "RemoveContainer" containerID="024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2" Nov 04 00:26:21.730727 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.730698 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768"] Nov 04 00:26:21.742276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.742252 6208 scope.go:122] "RemoveContainer" containerID="024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2" Nov 04 00:26:21.742781 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.742692 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-b8768"] Nov 04 00:26:21.743871 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:26:21.743844 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2\": container with ID starting with 024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2 not found: ID does not exist" containerID="024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2" Nov 04 00:26:21.744409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.744360 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2"} err="failed to get container status \"024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2\": rpc error: code = NotFound desc = could not find container \"024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2\": container with ID starting with 024a2f17d58fa416f3a5410befda4d82c43fe4be0c2bc88809fca098982291c2 not found: ID does not exist" Nov 04 00:26:21.751466 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.751443 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z"] Nov 04 00:26:21.758012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.757989 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85ad28-openshift-default-7dfd545fb9-8h57z"] Nov 04 00:26:21.916543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:21.916510 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:22.331716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:22.331683 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:23.214452 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:23.214414 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:23.319271 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:23.319237 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" path="/var/lib/kubelet/pods/515f512b-c7a2-4d2c-b443-bfe912c77f80/volumes" Nov 04 00:26:23.319830 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:23.319814 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" path="/var/lib/kubelet/pods/a1516c89-1bc2-4251-90f8-f1d6688dfdf6/volumes" Nov 04 00:26:24.030482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:24.030153 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:24.854550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:24.854514 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:25.718548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:25.718512 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:26.507406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:26.507371 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:27.417787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:27.417748 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:28.126061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:28.125992 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:28.958886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:28.958847 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:29.176261 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:29.176218 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:29.274375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:29.274286 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:29.939727 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:29.939687 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:30.690574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:30.690527 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:31.444407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:31.444365 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:32.263604 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:32.263566 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:33.022725 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:33.022686 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:33.745727 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:33.745675 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:34.517947 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:34.517907 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:35.224903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:35.224861 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:35.706937 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:26:35.706905 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:26:35.949725 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:35.949691 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:36.550028 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:36.549991 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:36.658237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:36.658202 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:37.388555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:37.388507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:38.245471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:38.245432 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:38.948599 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:38.948564 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:39.718419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:39.718385 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:40.448809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:40.448773 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:41.151466 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:41.151421 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:41.947483 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:41.947450 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:42.837069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:42.837025 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:43.334807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:43.334759 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:43.655337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:43.655249 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:44.531399 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:44.531362 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:45.403600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:45.403561 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:46.128323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:46.128284 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:46.860850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:46.860821 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:47.349094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:47.349047 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:47.474400 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:47.474367 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:47.672346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:47.672263 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:48.296915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:48.296872 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:48.762860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:48.762828 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:49.553777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:49.553741 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:50.445151 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:50.445112 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:51.235918 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:51.235874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:52.049335 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:52.049301 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:52.808852 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:52.808803 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:52.908913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:52.908877 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:53.012591 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:53.012552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:53.339617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:53.339575 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:53.723113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:53.684870 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n"] Nov 04 00:26:53.723876 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:53.723839 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" containerID="cri-o://50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93" gracePeriod=30 Nov 04 00:26:53.740510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:53.722662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:54.465246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:54.465210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:54.565488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:54.565451 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:55.323011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:55.322976 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:55.846624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:55.846587 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:56.045580 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:56.045537 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:56.748813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:56.748779 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:57.516465 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:57.516434 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:58.245554 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:58.245522 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:58.932946 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:58.932910 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:59.126553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.126516 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:26:59.559164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.559140 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:26:59.630495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.630462 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-certs\") pod \"1b1903fe-748c-4b83-9d58-7727b1f9b755\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " Nov 04 00:26:59.630681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.630503 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-podinfo\") pod \"1b1903fe-748c-4b83-9d58-7727b1f9b755\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " Nov 04 00:26:59.630681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.630594 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-token\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-token\") pod \"1b1903fe-748c-4b83-9d58-7727b1f9b755\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " Nov 04 00:26:59.630681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.630642 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-kube-api-access-dkfrc\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-kube-api-access-dkfrc\") pod \"1b1903fe-748c-4b83-9d58-7727b1f9b755\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " Nov 04 00:26:59.630681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.630678 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-envoy\") pod \"1b1903fe-748c-4b83-9d58-7727b1f9b755\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " Nov 04 00:26:59.630929 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.630790 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-socket\") pod \"1b1903fe-748c-4b83-9d58-7727b1f9b755\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " Nov 04 00:26:59.630929 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.630828 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-certs" pod "1b1903fe-748c-4b83-9d58-7727b1f9b755" (UID: "1b1903fe-748c-4b83-9d58-7727b1f9b755"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:59.630929 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.630908 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1903fe-748c-4b83-9d58-7727b1f9b755-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1903fe-748c-4b83-9d58-7727b1f9b755-istiod-ca-cert\") pod \"1b1903fe-748c-4b83-9d58-7727b1f9b755\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " Nov 04 00:26:59.631128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.630972 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-credential-socket\") pod \"1b1903fe-748c-4b83-9d58-7727b1f9b755\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " Nov 04 00:26:59.631128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.631054 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-data\") pod \"1b1903fe-748c-4b83-9d58-7727b1f9b755\" (UID: \"1b1903fe-748c-4b83-9d58-7727b1f9b755\") " Nov 04 00:26:59.631531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.631455 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-credential-socket" pod "1b1903fe-748c-4b83-9d58-7727b1f9b755" (UID: "1b1903fe-748c-4b83-9d58-7727b1f9b755"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:59.631651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.631532 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1903fe-748c-4b83-9d58-7727b1f9b755-istiod-ca-cert" pod "1b1903fe-748c-4b83-9d58-7727b1f9b755" (UID: "1b1903fe-748c-4b83-9d58-7727b1f9b755"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:26:59.631651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.631595 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-socket" pod "1b1903fe-748c-4b83-9d58-7727b1f9b755" (UID: "1b1903fe-748c-4b83-9d58-7727b1f9b755"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:59.631779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.631680 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-data" pod "1b1903fe-748c-4b83-9d58-7727b1f9b755" (UID: "1b1903fe-748c-4b83-9d58-7727b1f9b755"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:59.632164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.632071 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:59.632164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.632097 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b1903fe-748c-4b83-9d58-7727b1f9b755-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:59.632164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.632111 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:59.632164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.632126 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:59.632164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.632141 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:59.633501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.633458 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-podinfo" pod "1b1903fe-748c-4b83-9d58-7727b1f9b755" (UID: "1b1903fe-748c-4b83-9d58-7727b1f9b755"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:26:59.633813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.633789 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-kube-api-access-dkfrc" pod "1b1903fe-748c-4b83-9d58-7727b1f9b755" (UID: "1b1903fe-748c-4b83-9d58-7727b1f9b755"). InnerVolumeSpecName "kube-api-access-dkfrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:26:59.634063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.634012 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-token" pod "1b1903fe-748c-4b83-9d58-7727b1f9b755" (UID: "1b1903fe-748c-4b83-9d58-7727b1f9b755"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:26:59.634148 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.634016 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-envoy" pod "1b1903fe-748c-4b83-9d58-7727b1f9b755" (UID: "1b1903fe-748c-4b83-9d58-7727b1f9b755"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:26:59.733398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.733365 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:59.733398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.733394 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:59.733398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.733404 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkfrc\" (UniqueName: \"kubernetes.io/projected/1b1903fe-748c-4b83-9d58-7727b1f9b755-kube-api-access-dkfrc\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:59.733692 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.733414 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b1903fe-748c-4b83-9d58-7727b1f9b755-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:26:59.836440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:26:59.836359 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:00.246487 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.246448 6208 generic.go:356] "Generic (PLEG): container finished" podID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerID="50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93" exitCode=0 Nov 04 00:27:00.246955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.246506 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" Nov 04 00:27:00.246955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.246533 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" event={"ID":"1b1903fe-748c-4b83-9d58-7727b1f9b755","Type":"ContainerDied","Data":"50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93"} Nov 04 00:27:00.246955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.246616 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n" event={"ID":"1b1903fe-748c-4b83-9d58-7727b1f9b755","Type":"ContainerDied","Data":"e909317c8daf5981c491554a9eae67cdb6e9fbc5a021ed1b4d57f13e24abb585"} Nov 04 00:27:00.246955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.246644 6208 scope.go:122] "RemoveContainer" containerID="50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93" Nov 04 00:27:00.259297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.259276 6208 scope.go:122] "RemoveContainer" containerID="72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2" Nov 04 00:27:00.273007 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.272977 6208 scope.go:122] "RemoveContainer" containerID="50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93" Nov 04 00:27:00.273414 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:27:00.273388 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93\": container with ID starting with 50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93 not found: ID does not exist" containerID="50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93" Nov 04 00:27:00.273490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.273424 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93"} err="failed to get container status \"50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93\": rpc error: code = NotFound desc = could not find container \"50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93\": container with ID starting with 50b510693322f4c99b5d6047463aed2320af1f1460695c91f3b797c29a4ccd93 not found: ID does not exist" Nov 04 00:27:00.273490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.273451 6208 scope.go:122] "RemoveContainer" containerID="72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2" Nov 04 00:27:00.274230 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:27:00.273771 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2\": container with ID starting with 72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2 not found: ID does not exist" containerID="72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2" Nov 04 00:27:00.274230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.273801 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2"} err="failed to get container status \"72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2\": rpc error: code = NotFound desc = could not find container \"72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2\": container with ID starting with 72bc0417d276f4d41df3b49777ee45297116198c2d6a320df903060647faf8e2 not found: ID does not exist" Nov 04 00:27:00.275379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.275358 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n"] Nov 04 00:27:00.279144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.279124 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d94e94-openshift-default-6f46499c6f-lhb8n"] Nov 04 00:27:00.555625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:00.555536 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:01.135992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:01.135951 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:01.232135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:01.232094 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:01.367057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:01.355868 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" path="/var/lib/kubelet/pods/1b1903fe-748c-4b83-9d58-7727b1f9b755/volumes" Nov 04 00:27:01.639564 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:01.639524 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:02.219908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:02.219874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:02.445756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:02.445725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:03.160250 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:03.160204 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:03.627321 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:03.627286 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:03.973712 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:03.973678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:04.758890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:04.758855 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:05.538733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:05.538693 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:05.710438 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:27:05.710403 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:27:06.240841 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:06.240802 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:06.814744 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:06.814711 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt"] Nov 04 00:27:06.815260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:06.815229 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" podUID="a6081121-890a-43f4-b85f-2bc166d06035" containerName="maas-api" containerID="cri-o://61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2" gracePeriod=30 Nov 04 00:27:06.860019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:06.859985 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:07.331074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.331014 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:07.347646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.347618 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:27:07.382940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.382483 6208 generic.go:356] "Generic (PLEG): container finished" podID="a6081121-890a-43f4-b85f-2bc166d06035" containerID="61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2" exitCode=0 Nov 04 00:27:07.382940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.382633 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" event={"ID":"a6081121-890a-43f4-b85f-2bc166d06035","Type":"ContainerDied","Data":"61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2"} Nov 04 00:27:07.382940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.382698 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" event={"ID":"a6081121-890a-43f4-b85f-2bc166d06035","Type":"ContainerDied","Data":"856a799836e726ea46ebabf07d69c096f53fb2e9b48479f3a0afc7cf844e2c80"} Nov 04 00:27:07.382940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.382734 6208 scope.go:122] "RemoveContainer" containerID="61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2" Nov 04 00:27:07.383349 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.383076 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt" Nov 04 00:27:07.442674 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.442649 6208 scope.go:122] "RemoveContainer" containerID="61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2" Nov 04 00:27:07.443007 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:27:07.442983 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2\": container with ID starting with 61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2 not found: ID does not exist" containerID="61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2" Nov 04 00:27:07.443124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.443018 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2"} err="failed to get container status \"61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2\": rpc error: code = NotFound desc = could not find container \"61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2\": container with ID starting with 61241f550a91e7a1cc7afaea1225bb8d15c9ba50d65744028e458eaae683cbd2 not found: ID does not exist" Nov 04 00:27:07.513872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.513838 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-metrics-certs\") pod \"a6081121-890a-43f4-b85f-2bc166d06035\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " Nov 04 00:27:07.514093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.513953 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6081121-890a-43f4-b85f-2bc166d06035-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6081121-890a-43f4-b85f-2bc166d06035-openshift-service-ca\") pod \"a6081121-890a-43f4-b85f-2bc166d06035\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " Nov 04 00:27:07.514093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.514000 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-maas-api-tls\") pod \"a6081121-890a-43f4-b85f-2bc166d06035\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " Nov 04 00:27:07.514093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.514044 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6081121-890a-43f4-b85f-2bc166d06035-kube-api-access-vvq28\" (UniqueName: \"kubernetes.io/projected/a6081121-890a-43f4-b85f-2bc166d06035-kube-api-access-vvq28\") pod \"a6081121-890a-43f4-b85f-2bc166d06035\" (UID: \"a6081121-890a-43f4-b85f-2bc166d06035\") " Nov 04 00:27:07.514651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.514616 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6081121-890a-43f4-b85f-2bc166d06035-openshift-service-ca" pod "a6081121-890a-43f4-b85f-2bc166d06035" (UID: "a6081121-890a-43f4-b85f-2bc166d06035"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:27:07.514806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.514789 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6081121-890a-43f4-b85f-2bc166d06035-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:07.516574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.516540 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-maas-api-tls" pod "a6081121-890a-43f4-b85f-2bc166d06035" (UID: "a6081121-890a-43f4-b85f-2bc166d06035"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:27:07.517138 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.517113 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-metrics-certs" pod "a6081121-890a-43f4-b85f-2bc166d06035" (UID: "a6081121-890a-43f4-b85f-2bc166d06035"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:27:07.517238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.517148 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6081121-890a-43f4-b85f-2bc166d06035-kube-api-access-vvq28" pod "a6081121-890a-43f4-b85f-2bc166d06035" (UID: "a6081121-890a-43f4-b85f-2bc166d06035"). InnerVolumeSpecName "kube-api-access-vvq28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:27:07.616616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.616526 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:07.616616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.616561 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6081121-890a-43f4-b85f-2bc166d06035-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:07.616616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.616573 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvq28\" (UniqueName: \"kubernetes.io/projected/a6081121-890a-43f4-b85f-2bc166d06035-kube-api-access-vvq28\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:07.712155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.712116 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt"] Nov 04 00:27:07.715849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:07.715823 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-757a7d-5b44c6cffc-r25bt"] Nov 04 00:27:08.075956 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:08.075922 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:08.425570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:08.425481 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:08.527361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:08.526501 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:08.838096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:08.838023 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:08.947843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:08.947804 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:09.301679 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:09.301641 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6081121-890a-43f4-b85f-2bc166d06035" path="/var/lib/kubelet/pods/a6081121-890a-43f4-b85f-2bc166d06035/volumes" Nov 04 00:27:09.320819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:09.320786 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j"] Nov 04 00:27:09.321028 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:09.321004 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" containerID="cri-o://690cd326c8410dc4226d0ccef7aa0aa0fc7ac1f66f25507431ff26965c0821b2" gracePeriod=30 Nov 04 00:27:09.357996 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:09.357960 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:09.846864 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:09.846823 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:10.495447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:10.495415 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:10.591485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:10.591445 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:10.687965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:10.687919 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:10.954846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:10.954749 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:11.054323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:11.054262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:11.642737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:11.642696 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:12.623506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:12.623467 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:13.343448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:13.343405 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:14.049915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.049882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:14.445296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.445268 6208 generic.go:356] "Generic (PLEG): container finished" podID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerID="690cd326c8410dc4226d0ccef7aa0aa0fc7ac1f66f25507431ff26965c0821b2" exitCode=0 Nov 04 00:27:14.445427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.445344 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" event={"ID":"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd","Type":"ContainerDied","Data":"690cd326c8410dc4226d0ccef7aa0aa0fc7ac1f66f25507431ff26965c0821b2"} Nov 04 00:27:14.445427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.445408 6208 scope.go:122] "RemoveContainer" containerID="d9a516e3a4149a8d3202431d6c6fa72efbb5190e9065f212c72273b4270ec4e2" Nov 04 00:27:14.615705 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.615677 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:27:14.680938 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.680903 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-kube-api-access-85cdp\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-kube-api-access-85cdp\") pod \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " Nov 04 00:27:14.681174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.680963 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-socket\") pod \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " Nov 04 00:27:14.681174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.681071 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-token\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-token\") pod \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " Nov 04 00:27:14.681174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.681137 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-credential-socket\") pod \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " Nov 04 00:27:14.681298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.681171 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-podinfo\") pod \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " Nov 04 00:27:14.681298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.681210 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-certs\") pod \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " Nov 04 00:27:14.681298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.681250 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-envoy\") pod \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " Nov 04 00:27:14.681449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.681403 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istiod-ca-cert\") pod \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " Nov 04 00:27:14.682705 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.681546 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-data\") pod \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\" (UID: \"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd\") " Nov 04 00:27:14.682705 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.681610 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-credential-socket" pod "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" (UID: "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:27:14.682705 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.681832 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-socket" pod "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" (UID: "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:27:14.682705 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.682380 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:14.682705 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.682402 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:14.683575 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.683549 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-kube-api-access-85cdp" pod "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" (UID: "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd"). InnerVolumeSpecName "kube-api-access-85cdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:27:14.683897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.683869 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-data" pod "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" (UID: "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:27:14.684011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.683938 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-certs" pod "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" (UID: "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:27:14.684011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.683961 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istiod-ca-cert" pod "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" (UID: "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:27:14.684512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.684495 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-token" pod "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" (UID: "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:27:14.688069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.686171 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-podinfo" pod "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" (UID: "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:27:14.718617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.718586 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-envoy" pod "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" (UID: "49a4cc55-4497-4f36-aecd-cd7e5e21a0bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:27:14.783909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.783879 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:14.783909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.783907 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85cdp\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-kube-api-access-85cdp\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:14.784144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.783918 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:14.784144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.783927 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:14.784144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.783936 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:14.784144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.783944 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:14.784144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.783953 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:14.834918 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:14.834881 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:15.454224 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:15.454188 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" event={"ID":"49a4cc55-4497-4f36-aecd-cd7e5e21a0bd","Type":"ContainerDied","Data":"413c1446e8902c98a80a53f5cfee0e722318013fcf4931a2175d3ab886e67cd8"} Nov 04 00:27:15.454649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:15.454252 6208 scope.go:122] "RemoveContainer" containerID="690cd326c8410dc4226d0ccef7aa0aa0fc7ac1f66f25507431ff26965c0821b2" Nov 04 00:27:15.454649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:15.454195 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j" Nov 04 00:27:15.476813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:15.476788 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j"] Nov 04 00:27:15.482516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:15.482487 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-757a7d-openshift-default-59bcc54ff9-fvm5j"] Nov 04 00:27:15.544187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:15.544125 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:16.253418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:16.253386 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:17.083758 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:17.083717 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:17.298000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:17.297964 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" path="/var/lib/kubelet/pods/49a4cc55-4497-4f36-aecd-cd7e5e21a0bd/volumes" Nov 04 00:27:17.943566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:17.943528 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:18.798750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:18.798713 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:19.654194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:19.654150 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:20.535387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:20.535347 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:21.248339 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:21.248298 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:22.148064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:22.147988 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:23.027820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:23.027774 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:23.825474 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:23.825442 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:24.266983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:24.266945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:24.369307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:24.369259 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:24.943165 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:24.943117 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:25.447467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:25.447431 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:25.547049 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:25.547013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:25.939113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:25.939073 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:26.825972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:26.825935 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:27.647612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:27.647565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:28.459633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:28.459580 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77"] Nov 04 00:27:28.494076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:28.492011 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:28.591874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:28.591839 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:29.277796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:29.277755 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:30.052192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:30.052139 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:30.924355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:30.924314 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:31.830351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:31.830313 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:32.734269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:32.734230 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:33.470959 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:33.470922 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:34.221419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:34.221370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:35.036081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:35.036016 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:35.710209 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:27:35.710172 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:27:35.828634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:35.828598 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:36.555629 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:36.555582 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:37.329242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:37.329203 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:38.141829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:38.141764 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:38.840656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:38.840625 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:39.579722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:39.579666 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:39.867828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:39.867658 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:40.083472 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:40.083426 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:40.668538 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:40.668505 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:41.433328 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:41.433279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:42.227579 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:42.227546 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:42.341903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:42.341870 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:42.445667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:42.445577 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:42.569129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:42.569016 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:43.188737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.188702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:43.413521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.413485 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:43.510588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.510530 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" podUID="f73de605-a203-49e3-9401-7419f9effa52" containerName="main" containerID="cri-o://d161458d2a6bef68e629948d6d7e1072e3c876a13e27a334ad53e1929f6dc189" gracePeriod=45 Nov 04 00:27:43.515511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.515479 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:43.620633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.620580 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:43.690997 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.690951 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" podUID="f73de605-a203-49e3-9401-7419f9effa52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": dial tcp 10.133.0.188:8000: connect: connection refused" Nov 04 00:27:43.813695 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.813647 6208 generic.go:356] "Generic (PLEG): container finished" podID="f73de605-a203-49e3-9401-7419f9effa52" containerID="d161458d2a6bef68e629948d6d7e1072e3c876a13e27a334ad53e1929f6dc189" exitCode=0 Nov 04 00:27:43.813894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.813849 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" event={"ID":"f73de605-a203-49e3-9401-7419f9effa52","Type":"ContainerDied","Data":"d161458d2a6bef68e629948d6d7e1072e3c876a13e27a334ad53e1929f6dc189"} Nov 04 00:27:43.814029 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.813895 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" event={"ID":"f73de605-a203-49e3-9401-7419f9effa52","Type":"ContainerDied","Data":"f872c4386a2dc8e39936a1373483ae50c627c57920aef83248e6aa0134b2b9e6"} Nov 04 00:27:43.814029 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.813913 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f872c4386a2dc8e39936a1373483ae50c627c57920aef83248e6aa0134b2b9e6" Nov 04 00:27:43.824081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.824020 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:27:43.928766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.928726 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-home\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-home\") pod \"f73de605-a203-49e3-9401-7419f9effa52\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " Nov 04 00:27:43.928955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.928858 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-model-cache\") pod \"f73de605-a203-49e3-9401-7419f9effa52\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " Nov 04 00:27:43.928955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.928929 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-dshm\") pod \"f73de605-a203-49e3-9401-7419f9effa52\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " Nov 04 00:27:43.929095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.928988 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73de605-a203-49e3-9401-7419f9effa52-kube-api-access-dzwzt\" (UniqueName: \"kubernetes.io/projected/f73de605-a203-49e3-9401-7419f9effa52-kube-api-access-dzwzt\") pod \"f73de605-a203-49e3-9401-7419f9effa52\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " Nov 04 00:27:43.929095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.928991 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-home" pod "f73de605-a203-49e3-9401-7419f9effa52" (UID: "f73de605-a203-49e3-9401-7419f9effa52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:27:43.929215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.929188 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-model-cache" pod "f73de605-a203-49e3-9401-7419f9effa52" (UID: "f73de605-a203-49e3-9401-7419f9effa52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:27:43.929274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.929217 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73de605-a203-49e3-9401-7419f9effa52-tls-certs\" (UniqueName: \"kubernetes.io/secret/f73de605-a203-49e3-9401-7419f9effa52-tls-certs\") pod \"f73de605-a203-49e3-9401-7419f9effa52\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " Nov 04 00:27:43.929343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.929294 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-tmp-dir\") pod \"f73de605-a203-49e3-9401-7419f9effa52\" (UID: \"f73de605-a203-49e3-9401-7419f9effa52\") " Nov 04 00:27:43.929568 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.929545 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-tmp-dir" pod "f73de605-a203-49e3-9401-7419f9effa52" (UID: "f73de605-a203-49e3-9401-7419f9effa52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:27:43.929870 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.929851 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:43.929870 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.929870 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:43.930019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.929879 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:43.931256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.931228 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-dshm" pod "f73de605-a203-49e3-9401-7419f9effa52" (UID: "f73de605-a203-49e3-9401-7419f9effa52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:27:43.931348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.931285 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73de605-a203-49e3-9401-7419f9effa52-kube-api-access-dzwzt" pod "f73de605-a203-49e3-9401-7419f9effa52" (UID: "f73de605-a203-49e3-9401-7419f9effa52"). InnerVolumeSpecName "kube-api-access-dzwzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:27:43.931348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:43.931297 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73de605-a203-49e3-9401-7419f9effa52-tls-certs" pod "f73de605-a203-49e3-9401-7419f9effa52" (UID: "f73de605-a203-49e3-9401-7419f9effa52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:27:44.031440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:44.031342 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f73de605-a203-49e3-9401-7419f9effa52-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:44.031440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:44.031381 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f73de605-a203-49e3-9401-7419f9effa52-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:44.031440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:44.031395 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwzt\" (UniqueName: \"kubernetes.io/projected/f73de605-a203-49e3-9401-7419f9effa52-kube-api-access-dzwzt\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:27:44.345276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:44.345182 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:44.822069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:44.822016 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77" Nov 04 00:27:44.848237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:44.848203 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77"] Nov 04 00:27:44.852137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:44.852112 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ecd46b/test-model-ecd46b-kserve-75bb8c799f-r5f77"] Nov 04 00:27:45.144840 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:45.144755 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:45.342932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:45.342886 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73de605-a203-49e3-9401-7419f9effa52" path="/var/lib/kubelet/pods/f73de605-a203-49e3-9401-7419f9effa52/volumes" Nov 04 00:27:45.845285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:45.845243 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:45.945570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:45.945533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:46.050617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:46.050576 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:46.449334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:46.449300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:47.347434 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:47.347393 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:47.463602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:47.463566 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:47.576434 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:47.576399 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:47.922954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:47.922916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:48.342867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:48.331594 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:49.226847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:49.226810 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:49.956979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:49.956943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:50.745395 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:50.745356 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:51.462079 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:51.462051 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:52.252446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:52.252410 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:53.008380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:53.008345 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:53.777982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:53.777945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:54.455318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:54.455284 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:54.572534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:54.572497 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:55.269286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:55.269228 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:55.524877 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:55.524794 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:56.337671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:56.337629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:57.445488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:57.445456 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:58.410395 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:58.410361 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:59.028638 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:59.028593 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:27:59.354089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:27:59.353986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:00.142935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:00.142900 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:00.917849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:00.917810 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:01.625701 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:01.625668 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:02.382222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:02.382188 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:02.482284 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:02.482246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:03.165887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:03.165846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:03.922327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:03.922291 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:04.719762 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:04.719720 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:05.433945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:05.433904 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:05.705872 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:28:05.705835 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:28:06.145954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:06.145856 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:06.929234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:06.929196 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:07.672843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:07.672805 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:08.480086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:08.480002 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:09.321322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:09.321262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:10.024385 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:10.024348 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:10.877434 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:10.877358 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:11.402086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:11.402048 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:11.502409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:11.502374 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:11.822334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:11.822302 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:11.922683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:11.922636 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:12.304264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:12.304228 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw"] Nov 04 00:28:12.304508 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:12.304479 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" containerID="cri-o://16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084" gracePeriod=30 Nov 04 00:28:12.341693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:12.341658 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:12.848751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:12.848711 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:13.137884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:13.137793 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:13.712307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:13.712269 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:14.622950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:14.622913 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:15.332433 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:15.332394 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:15.449056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:15.440717 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:17.146323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.146282 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:17.603856 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.603827 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:28:17.722557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.722526 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-certs\") pod \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " Nov 04 00:28:17.722739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.722566 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-socket\") pod \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " Nov 04 00:28:17.722739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.722725 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-kube-api-access-7t2nt\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-kube-api-access-7t2nt\") pod \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " Nov 04 00:28:17.722865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.722757 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-envoy\") pod \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " Nov 04 00:28:17.722865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.722788 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-podinfo\") pod \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " Nov 04 00:28:17.722865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.722834 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-certs" pod "8a2762e9-da42-47f7-b645-6dc0233f3f8d" (UID: "8a2762e9-da42-47f7-b645-6dc0233f3f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:28:17.723707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.722876 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-token\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-token\") pod \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " Nov 04 00:28:17.723707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.722925 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-socket" pod "8a2762e9-da42-47f7-b645-6dc0233f3f8d" (UID: "8a2762e9-da42-47f7-b645-6dc0233f3f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:28:17.723707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.722981 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istiod-ca-cert\") pod \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " Nov 04 00:28:17.723707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723011 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-credential-socket\") pod \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " Nov 04 00:28:17.723707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723170 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-data\") pod \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\" (UID: \"8a2762e9-da42-47f7-b645-6dc0233f3f8d\") " Nov 04 00:28:17.723707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723302 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-credential-socket" pod "8a2762e9-da42-47f7-b645-6dc0233f3f8d" (UID: "8a2762e9-da42-47f7-b645-6dc0233f3f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:28:17.723707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723370 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istiod-ca-cert" pod "8a2762e9-da42-47f7-b645-6dc0233f3f8d" (UID: "8a2762e9-da42-47f7-b645-6dc0233f3f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:28:17.723707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723568 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-data" pod "8a2762e9-da42-47f7-b645-6dc0233f3f8d" (UID: "8a2762e9-da42-47f7-b645-6dc0233f3f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:28:17.724232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723906 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:17.724232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723930 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:17.724232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723946 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:17.724232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723959 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:17.724232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.723971 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:17.725455 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.725428 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-kube-api-access-7t2nt" pod "8a2762e9-da42-47f7-b645-6dc0233f3f8d" (UID: "8a2762e9-da42-47f7-b645-6dc0233f3f8d"). InnerVolumeSpecName "kube-api-access-7t2nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:28:17.725601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.725466 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-token" pod "8a2762e9-da42-47f7-b645-6dc0233f3f8d" (UID: "8a2762e9-da42-47f7-b645-6dc0233f3f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:28:17.725810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.725783 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-podinfo" pod "8a2762e9-da42-47f7-b645-6dc0233f3f8d" (UID: "8a2762e9-da42-47f7-b645-6dc0233f3f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:28:17.726336 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.726311 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-envoy" pod "8a2762e9-da42-47f7-b645-6dc0233f3f8d" (UID: "8a2762e9-da42-47f7-b645-6dc0233f3f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:28:17.824943 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.824911 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:17.824943 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.824938 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2nt\" (UniqueName: \"kubernetes.io/projected/8a2762e9-da42-47f7-b645-6dc0233f3f8d-kube-api-access-7t2nt\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:17.824943 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.824948 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:17.825215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.824959 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a2762e9-da42-47f7-b645-6dc0233f3f8d-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:17.899405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:17.899370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:18.249550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.249515 6208 generic.go:356] "Generic (PLEG): container finished" podID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerID="16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084" exitCode=0 Nov 04 00:28:18.249955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.249565 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" Nov 04 00:28:18.249955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.249607 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" event={"ID":"8a2762e9-da42-47f7-b645-6dc0233f3f8d","Type":"ContainerDied","Data":"16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084"} Nov 04 00:28:18.249955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.249666 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw" event={"ID":"8a2762e9-da42-47f7-b645-6dc0233f3f8d","Type":"ContainerDied","Data":"e52d0a0266daaee6aaf16a6f095f11e407ade498fabaa06cd911300b99019e9b"} Nov 04 00:28:18.249955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.249696 6208 scope.go:122] "RemoveContainer" containerID="16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084" Nov 04 00:28:18.262472 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.262448 6208 scope.go:122] "RemoveContainer" containerID="cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc" Nov 04 00:28:18.274165 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.274145 6208 scope.go:122] "RemoveContainer" containerID="16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084" Nov 04 00:28:18.274445 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:28:18.274424 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084\": container with ID starting with 16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084 not found: ID does not exist" containerID="16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084" Nov 04 00:28:18.274509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.274454 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084"} err="failed to get container status \"16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084\": rpc error: code = NotFound desc = could not find container \"16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084\": container with ID starting with 16fa8fe5fcaf0d00daba960df0a93110e7946be9e2db521486f922d8a46a1084 not found: ID does not exist" Nov 04 00:28:18.274509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.274475 6208 scope.go:122] "RemoveContainer" containerID="cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc" Nov 04 00:28:18.274743 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:28:18.274724 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc\": container with ID starting with cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc not found: ID does not exist" containerID="cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc" Nov 04 00:28:18.274792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.274752 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc"} err="failed to get container status \"cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc\": rpc error: code = NotFound desc = could not find container \"cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc\": container with ID starting with cb2ccf78fbc11d337602d787d2bdb876d4c22a84df6e93eb23e4d9a90a5049cc not found: ID does not exist" Nov 04 00:28:18.277333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.277310 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw"] Nov 04 00:28:18.280820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.280799 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ecd46b-openshift-default-7cbc8d464c-dwzsw"] Nov 04 00:28:18.738919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:18.738884 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:19.299869 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:19.299832 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" path="/var/lib/kubelet/pods/8a2762e9-da42-47f7-b645-6dc0233f3f8d/volumes" Nov 04 00:28:19.525361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:19.525325 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:20.216965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:20.212291 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn"] Nov 04 00:28:20.222661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:20.222631 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:20.319066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:20.319021 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:20.420581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:20.420544 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:21.167345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:21.167309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:21.301414 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:21.301383 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:22.047599 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:22.047561 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:22.831720 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:22.831687 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:23.554683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:23.554650 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:24.340806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:24.340769 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:25.123747 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:25.123713 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:25.214164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:25.214110 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" podUID="3974cd05-5e4b-4bb8-973a-648db4082546" containerName="payload-pre-processing" containerID="cri-o://e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c" gracePeriod=25 Nov 04 00:28:25.842015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:25.841977 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:26.676801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:26.676762 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:26.859360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:26.859321 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:27.539426 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:27.539385 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:27.967471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:27.967441 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:28.319967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:28.319870 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:29.044385 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:29.044344 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:29.815698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:29.815665 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:30.573766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:30.573720 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:31.351488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:31.351452 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:31.939517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:31.939483 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:33.362077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:33.362022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:34.223090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:34.216943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:34.327061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:34.319055 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:34.429746 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:34.429696 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:34.465498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:34.464249 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" podUID="3974cd05-5e4b-4bb8-973a-648db4082546" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:28:34.681105 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:34.676095 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:35.028714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:35.028653 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:35.328366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:35.326628 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:35.646003 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:35.645927 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:35.728613 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:28:35.728572 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:28:36.445967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:36.445925 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:36.732656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:36.732606 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:36.837316 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:36.837272 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:37.217101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:37.217063 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:37.492445 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:37.492302 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:37.716998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:37.716948 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:38.418835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:38.418800 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:39.214189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:39.214140 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:39.949940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:39.949905 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:40.737858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:40.737817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:41.431535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:41.431496 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:42.153559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:42.153506 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:42.935533 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:42.935500 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:43.465351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:43.465307 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:43.824897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:43.824809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:44.447819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:44.447779 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" podUID="3974cd05-5e4b-4bb8-973a-648db4082546" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:28:44.547304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:44.547265 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:45.314226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.314193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:45.532755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.532721 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:28:45.626385 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.626351 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3974cd05-5e4b-4bb8-973a-648db4082546-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3974cd05-5e4b-4bb8-973a-648db4082546-plugins-config-volume\") pod \"3974cd05-5e4b-4bb8-973a-648db4082546\" (UID: \"3974cd05-5e4b-4bb8-973a-648db4082546\") " Nov 04 00:28:45.626571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.626457 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3974cd05-5e4b-4bb8-973a-648db4082546-kube-api-access-jlfjm\" (UniqueName: \"kubernetes.io/projected/3974cd05-5e4b-4bb8-973a-648db4082546-kube-api-access-jlfjm\") pod \"3974cd05-5e4b-4bb8-973a-648db4082546\" (UID: \"3974cd05-5e4b-4bb8-973a-648db4082546\") " Nov 04 00:28:45.626740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.626716 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3974cd05-5e4b-4bb8-973a-648db4082546-plugins-config-volume" pod "3974cd05-5e4b-4bb8-973a-648db4082546" (UID: "3974cd05-5e4b-4bb8-973a-648db4082546"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:28:45.627045 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.627015 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3974cd05-5e4b-4bb8-973a-648db4082546-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:45.628519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.628497 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3974cd05-5e4b-4bb8-973a-648db4082546-kube-api-access-jlfjm" pod "3974cd05-5e4b-4bb8-973a-648db4082546" (UID: "3974cd05-5e4b-4bb8-973a-648db4082546"). InnerVolumeSpecName "kube-api-access-jlfjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:28:45.676506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.676470 6208 generic.go:356] "Generic (PLEG): container finished" podID="3974cd05-5e4b-4bb8-973a-648db4082546" containerID="e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c" exitCode=0 Nov 04 00:28:45.676665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.676520 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" Nov 04 00:28:45.676665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.676543 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" event={"ID":"3974cd05-5e4b-4bb8-973a-648db4082546","Type":"ContainerDied","Data":"e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c"} Nov 04 00:28:45.676665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.676621 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn" event={"ID":"3974cd05-5e4b-4bb8-973a-648db4082546","Type":"ContainerDied","Data":"7da6a45e6c56e674379c4ce1f46d316a204f633afefdd9a393ecbc42fb50a8f2"} Nov 04 00:28:45.676665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.676643 6208 scope.go:122] "RemoveContainer" containerID="e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c" Nov 04 00:28:45.687994 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.687976 6208 scope.go:122] "RemoveContainer" containerID="e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c" Nov 04 00:28:45.688293 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:28:45.688273 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c\": container with ID starting with e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c not found: ID does not exist" containerID="e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c" Nov 04 00:28:45.688372 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.688302 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c"} err="failed to get container status \"e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c\": rpc error: code = NotFound desc = could not find container \"e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c\": container with ID starting with e8893b83f19c6f7f1f690beb5a330122b30d651c671d62534a73dd903774ee6c not found: ID does not exist" Nov 04 00:28:45.702683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.702652 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn"] Nov 04 00:28:45.707301 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.707269 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6382fc-6f6bd4f88c-9swtn"] Nov 04 00:28:45.728175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:45.728146 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlfjm\" (UniqueName: \"kubernetes.io/projected/3974cd05-5e4b-4bb8-973a-648db4082546-kube-api-access-jlfjm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:28:46.017200 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:46.017167 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:46.439896 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:46.439819 6208 scope.go:122] "RemoveContainer" containerID="d161458d2a6bef68e629948d6d7e1072e3c876a13e27a334ad53e1929f6dc189" Nov 04 00:28:46.850190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:46.850153 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:47.295186 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:47.295152 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3974cd05-5e4b-4bb8-973a-648db4082546" path="/var/lib/kubelet/pods/3974cd05-5e4b-4bb8-973a-648db4082546/volumes" Nov 04 00:28:47.497710 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:47.497668 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:47.616092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:47.599480 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:48.364489 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:48.364451 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:49.085928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:49.085890 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:49.847214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:49.847182 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:50.547317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:50.547278 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:51.267256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:51.267205 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:52.037728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:52.037678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:52.876533 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:52.876503 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:53.626056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:53.624890 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:54.342009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:54.341941 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:55.117059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:55.117006 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:55.926407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:55.926367 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:56.735549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:56.735508 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:57.635326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:57.635286 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:58.518252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:58.518217 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:59.088820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:59.088777 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:28:59.309560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:28:59.309522 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:00.024006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:00.023970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:00.854968 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:00.854928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:01.219488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:01.219449 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:01.508070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:01.507969 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:02.329403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:02.329366 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:03.038523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:03.038484 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:03.737677 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:03.737642 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:04.577697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:04.577660 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:05.450860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:05.450829 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:05.715671 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:29:05.715630 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:29:06.454073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:06.449005 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:07.239947 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:07.239912 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:08.059225 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:08.059187 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:08.929232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:08.929200 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:09.630282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:09.630250 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:10.313519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:10.313479 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:10.454465 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:10.454431 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:11.343315 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:11.343190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:12.045640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:12.045608 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:12.743718 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:12.743685 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:13.535131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:13.535094 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:14.312370 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:14.312334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:15.122042 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:15.122006 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:15.823175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:15.823143 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:16.547967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:16.547927 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:17.364628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:17.364586 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:18.220304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:18.220258 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:19.121046 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:19.120994 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:19.826433 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:19.826395 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:20.106883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:20.106797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:20.537318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:20.537277 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:21.251844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:21.251806 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:22.023817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:22.023785 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:22.837551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:22.837513 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:23.548279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:23.548237 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:24.317312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:24.317276 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:25.108835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:25.108797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:25.868287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:25.868248 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:26.640522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:26.640479 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:27.434624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:27.434590 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:28.226015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:28.225957 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:29.245674 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:29.245639 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:30.131583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:30.131548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:30.840184 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:30.840142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:31.547666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:31.547624 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:32.336640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:32.336604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:33.060419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:33.060383 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:33.849515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:33.849483 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:34.624985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:34.624943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:35.520611 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:35.520580 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:35.705443 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:29:35.705407 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:29:36.235375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:36.235320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:36.482782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:36.482734 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:36.952203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:36.952163 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:37.849228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:37.849189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:38.070011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:38.069957 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:38.180843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:38.180762 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:38.591059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:38.590998 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:39.386116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:39.385832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:40.026939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:40.026908 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:40.745100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:40.745066 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:41.543578 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:41.543543 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:42.254193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:42.254158 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:42.967362 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:42.967335 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:29:42.969237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:42.969210 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:29:43.039481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:43.039444 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:43.533858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:43.533824 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:43.754265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:43.754228 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:44.589912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:44.589873 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:44.701089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:44.700560 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:45.318013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:45.317979 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:46.118136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:46.118089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:46.853657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:46.853622 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:47.554519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:47.554479 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:47.672057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:47.667189 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:48.410638 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:48.410589 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:48.511192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:48.511157 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:49.147150 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:49.147117 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:49.868411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:49.868332 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:50.650492 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:50.650452 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:51.427654 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:51.427618 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:52.118772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:52.118739 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:52.850526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:52.850488 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:53.833055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:53.829659 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:55.023540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:55.023507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:55.672609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:55.672565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:55.929706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:55.929626 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:56.718518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:56.718458 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:57.432763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:57.432723 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:57.662222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:57.662185 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:58.324506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:58.324462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:59.042608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:59.042574 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:59.156908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:59.156874 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:29:59.812622 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:29:59.812582 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:00.206176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.206140 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95"] Nov 04 00:30:00.207369 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207340 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" Nov 04 00:30:00.207577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207552 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207587 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207596 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207609 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a6081121-890a-43f4-b85f-2bc166d06035" containerName="maas-api" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207618 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6081121-890a-43f4-b85f-2bc166d06035" containerName="maas-api" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207634 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207642 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207658 6208 cpu_manager.go:414] "containerMap: removing container" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207665 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207678 6208 cpu_manager.go:414] "containerMap: removing container" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207686 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207696 6208 cpu_manager.go:414] "containerMap: removing container" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207705 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207714 6208 cpu_manager.go:414] "containerMap: removing container" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207721 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207733 6208 cpu_manager.go:414] "containerMap: removing container" podUID="28b0b70f-ad5e-47e5-af92-f743b51805ad" containerName="maas-api" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207741 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b0b70f-ad5e-47e5-af92-f743b51805ad" containerName="maas-api" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207752 6208 cpu_manager.go:414] "containerMap: removing container" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207759 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207784 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f73de605-a203-49e3-9401-7419f9effa52" containerName="main" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207791 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73de605-a203-49e3-9401-7419f9effa52" containerName="main" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207801 6208 cpu_manager.go:414] "containerMap: removing container" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207808 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207820 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207828 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207842 6208 cpu_manager.go:414] "containerMap: removing container" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207850 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207865 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207872 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" Nov 04 00:30:00.207868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207883 6208 cpu_manager.go:414] "containerMap: removing container" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207890 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207910 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207917 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207931 6208 cpu_manager.go:414] "containerMap: removing container" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207938 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207951 6208 cpu_manager.go:414] "containerMap: removing container" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207960 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207980 6208 cpu_manager.go:414] "containerMap: removing container" podUID="46d8c34f-fd93-4700-8430-f75fcc6a670f" containerName="maas-api" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207987 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d8c34f-fd93-4700-8430-f75fcc6a670f" containerName="maas-api" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.207996 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3974cd05-5e4b-4bb8-973a-648db4082546" containerName="payload-pre-processing" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208002 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3974cd05-5e4b-4bb8-973a-648db4082546" containerName="payload-pre-processing" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208018 6208 cpu_manager.go:414] "containerMap: removing container" podUID="71e0e69d-ea9d-4bf1-b4cb-14c10731a974" containerName="maas-api" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208025 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e0e69d-ea9d-4bf1-b4cb-14c10731a974" containerName="maas-api" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208077 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208085 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208097 6208 cpu_manager.go:414] "containerMap: removing container" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208104 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208301 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d8c34f-fd93-4700-8430-f75fcc6a670f" containerName="maas-api" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208314 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208323 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3974cd05-5e4b-4bb8-973a-648db4082546" containerName="payload-pre-processing" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208333 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e0e69d-ea9d-4bf1-b4cb-14c10731a974" containerName="maas-api" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208343 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e32687-55f2-4eb8-bf9d-af860aa9848e" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208353 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208364 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4526c8ef-a23a-492b-99cf-31957ea71f0f" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208373 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208382 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6081121-890a-43f4-b85f-2bc166d06035" containerName="maas-api" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208391 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb44f32-c801-4a3b-b97b-81e776ddbbd6" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208405 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208415 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208424 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d797191-b9e1-4fc7-88bb-4eda431424b4" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208434 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="8390a996-22ca-4972-90d7-c2c6f104c4cc" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208442 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b0b70f-ad5e-47e5-af92-f743b51805ad" containerName="maas-api" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208453 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="515f512b-c7a2-4d2c-b443-bfe912c77f80" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208465 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73de605-a203-49e3-9401-7419f9effa52" containerName="main" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208474 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208482 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208492 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1516c89-1bc2-4251-90f8-f1d6688dfdf6" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208500 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208509 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208522 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2762e9-da42-47f7-b645-6dc0233f3f8d" containerName="istio-proxy" Nov 04 00:30:00.209772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.208533 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" Nov 04 00:30:00.214007 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.213718 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7"] Nov 04 00:30:00.214093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.214017 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:00.218344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218312 6208 cpu_manager.go:414] "containerMap: removing container" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" Nov 04 00:30:00.218344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218343 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257b95e5-b863-4444-9a35-efd3ca15e157" containerName="istio-proxy" Nov 04 00:30:00.218601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218388 6208 cpu_manager.go:414] "containerMap: removing container" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" Nov 04 00:30:00.218601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218396 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" Nov 04 00:30:00.218601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218409 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" Nov 04 00:30:00.218601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218417 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1156db56-b84d-4241-9689-69a6aa1eaf5e" containerName="istio-proxy" Nov 04 00:30:00.218601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218449 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" Nov 04 00:30:00.218601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218456 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" Nov 04 00:30:00.218934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218910 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1903fe-748c-4b83-9d58-7727b1f9b755" containerName="istio-proxy" Nov 04 00:30:00.218990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218942 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7485bb-2460-4c40-8815-bcfaf0a2c9a0" containerName="istio-proxy" Nov 04 00:30:00.218990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218957 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="555966c0-f453-4441-8991-cf25c1200236" containerName="istio-proxy" Nov 04 00:30:00.218990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.218971 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a4cc55-4497-4f36-aecd-cd7e5e21a0bd" containerName="istio-proxy" Nov 04 00:30:00.226987 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.226963 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:00.227549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.227516 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95"] Nov 04 00:30:00.234722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.234650 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7"] Nov 04 00:30:00.247415 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.242567 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws"] Nov 04 00:30:00.248085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.248016 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:00.256082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.256057 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt"] Nov 04 00:30:00.279101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.278421 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\"" Nov 04 00:30:00.283669 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.283637 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:00.294587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.294555 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws"] Nov 04 00:30:00.302380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.302353 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt"] Nov 04 00:30:00.318435 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.318405 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp"] Nov 04 00:30:00.325416 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.325392 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:00.327424 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.327397 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp"] Nov 04 00:30:00.341268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.337205 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp"] Nov 04 00:30:00.341268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.339221 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:00.354656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.354536 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp"] Nov 04 00:30:00.359534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.359197 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9c8m\" (UniqueName: \"kubernetes.io/projected/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-kube-api-access-l9c8m\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws\" (UID: \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:00.359534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.359318 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a17c-4f5d-461d-a1a1-c08bbd170078-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895870-24sj7\" (UID: \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:00.359534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.359451 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws\" (UID: \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:00.360080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.359545 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7skrj\" (UniqueName: \"kubernetes.io/projected/49e453cb-0248-4842-936e-c1e2716d822c-kube-api-access-7skrj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95\" (UID: \"49e453cb-0248-4842-936e-c1e2716d822c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:00.360080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.359653 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krh6c\" (UniqueName: \"kubernetes.io/projected/3f24a17c-4f5d-461d-a1a1-c08bbd170078-kube-api-access-krh6c\") pod \"maas-api-key-cleanup-29895870-24sj7\" (UID: \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:00.360080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.359697 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e453cb-0248-4842-936e-c1e2716d822c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95\" (UID: \"49e453cb-0248-4842-936e-c1e2716d822c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:00.462025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.461931 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a17c-4f5d-461d-a1a1-c08bbd170078-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895870-24sj7\" (UID: \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:00.462025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.461996 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msznv\" (UniqueName: \"kubernetes.io/projected/9946a8b2-7f05-48c3-9818-4badb91f171e-kube-api-access-msznv\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp\" (UID: \"9946a8b2-7f05-48c3-9818-4badb91f171e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:00.462276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462169 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws\" (UID: \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:00.462276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462258 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drtpp\" (UniqueName: \"kubernetes.io/projected/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-kube-api-access-drtpp\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt\" (UID: \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:00.463080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462321 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7skrj\" (UniqueName: \"kubernetes.io/projected/49e453cb-0248-4842-936e-c1e2716d822c-kube-api-access-7skrj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95\" (UID: \"49e453cb-0248-4842-936e-c1e2716d822c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:00.463080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462425 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt\" (UID: \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:00.463080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462522 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krh6c\" (UniqueName: \"kubernetes.io/projected/3f24a17c-4f5d-461d-a1a1-c08bbd170078-kube-api-access-krh6c\") pod \"maas-api-key-cleanup-29895870-24sj7\" (UID: \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:00.463080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462564 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e453cb-0248-4842-936e-c1e2716d822c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95\" (UID: \"49e453cb-0248-4842-936e-c1e2716d822c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:00.463080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462813 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a17c-4f5d-461d-a1a1-c08bbd170078-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895870-24sj7\" (UID: \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:00.463080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462864 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fms\" (UniqueName: \"kubernetes.io/projected/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-kube-api-access-56fms\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp\" (UID: \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:00.463080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462892 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp\" (UID: \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:00.463080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462914 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946a8b2-7f05-48c3-9818-4badb91f171e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp\" (UID: \"9946a8b2-7f05-48c3-9818-4badb91f171e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:00.463080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.462957 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws\" (UID: \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:00.463459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.463110 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9c8m\" (UniqueName: \"kubernetes.io/projected/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-kube-api-access-l9c8m\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws\" (UID: \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:00.463459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.463198 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e453cb-0248-4842-936e-c1e2716d822c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95\" (UID: \"49e453cb-0248-4842-936e-c1e2716d822c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:00.475363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.475337 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7skrj\" (UniqueName: \"kubernetes.io/projected/49e453cb-0248-4842-936e-c1e2716d822c-kube-api-access-7skrj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95\" (UID: \"49e453cb-0248-4842-936e-c1e2716d822c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:00.475667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.475643 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9c8m\" (UniqueName: \"kubernetes.io/projected/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-kube-api-access-l9c8m\") pod \"maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws\" (UID: \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:00.475713 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.475643 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krh6c\" (UniqueName: \"kubernetes.io/projected/3f24a17c-4f5d-461d-a1a1-c08bbd170078-kube-api-access-krh6c\") pod \"maas-api-key-cleanup-29895870-24sj7\" (UID: \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:00.556824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.556787 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:00.564834 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.564798 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drtpp\" (UniqueName: \"kubernetes.io/projected/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-kube-api-access-drtpp\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt\" (UID: \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:00.564972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.564868 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt\" (UID: \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:00.564972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.564926 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fms\" (UniqueName: \"kubernetes.io/projected/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-kube-api-access-56fms\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp\" (UID: \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:00.564972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.564962 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp\" (UID: \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:00.565146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.564989 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946a8b2-7f05-48c3-9818-4badb91f171e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp\" (UID: \"9946a8b2-7f05-48c3-9818-4badb91f171e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:00.565211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.565175 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msznv\" (UniqueName: \"kubernetes.io/projected/9946a8b2-7f05-48c3-9818-4badb91f171e-kube-api-access-msznv\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp\" (UID: \"9946a8b2-7f05-48c3-9818-4badb91f171e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:00.565589 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.565569 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt\" (UID: \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:00.565786 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.565762 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp\" (UID: \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:00.565841 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.565810 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946a8b2-7f05-48c3-9818-4badb91f171e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp\" (UID: \"9946a8b2-7f05-48c3-9818-4badb91f171e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:00.577343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.577316 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drtpp\" (UniqueName: \"kubernetes.io/projected/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-kube-api-access-drtpp\") pod \"maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt\" (UID: \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:00.577643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.577619 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msznv\" (UniqueName: \"kubernetes.io/projected/9946a8b2-7f05-48c3-9818-4badb91f171e-kube-api-access-msznv\") pod \"maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp\" (UID: \"9946a8b2-7f05-48c3-9818-4badb91f171e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:00.577744 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.577726 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fms\" (UniqueName: \"kubernetes.io/projected/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-kube-api-access-56fms\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp\" (UID: \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:00.618521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.618484 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:00.655232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.633885 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:00.664719 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.664674 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:00.674012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.673422 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:00.682807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.682602 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:00.775108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.774152 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:00.852906 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.852741 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95"] Nov 04 00:30:00.877758 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:30:00.877720 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e453cb_0248_4842_936e_c1e2716d822c.slice/crio-7c4acdb32898d1944202fb28fa480dece646befbbf9fa28eeb466c912b9a15c4 WatchSource:0}: Error finding container 7c4acdb32898d1944202fb28fa480dece646befbbf9fa28eeb466c912b9a15c4: Status 404 returned error can't find the container with id 7c4acdb32898d1944202fb28fa480dece646befbbf9fa28eeb466c912b9a15c4 Nov 04 00:30:00.973447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:00.973407 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7"] Nov 04 00:30:01.016617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.016582 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws"] Nov 04 00:30:01.056930 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:30:01.056888 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f24a17c_4f5d_461d_a1a1_c08bbd170078.slice/crio-13d89b2ce029cc98e7782a033977695716af0620bce850939ab0ada8f19d9966 WatchSource:0}: Error finding container 13d89b2ce029cc98e7782a033977695716af0620bce850939ab0ada8f19d9966: Status 404 returned error can't find the container with id 13d89b2ce029cc98e7782a033977695716af0620bce850939ab0ada8f19d9966 Nov 04 00:30:01.057581 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:30:01.057555 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89632be3_5d3a_4b2f_b51c_7f6b21395b5a.slice/crio-769038ea1112676e8ead4a7d50f272e03b0ccc798e45e44d677371c2502f4b1a WatchSource:0}: Error finding container 769038ea1112676e8ead4a7d50f272e03b0ccc798e45e44d677371c2502f4b1a: Status 404 returned error can't find the container with id 769038ea1112676e8ead4a7d50f272e03b0ccc798e45e44d677371c2502f4b1a Nov 04 00:30:01.076200 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.076170 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt"] Nov 04 00:30:01.078345 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:30:01.078288 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb1cff7_28c2_40e7_8dda_da12eb866ca6.slice/crio-0768b1ca5823ef676103b172c0360ef61b499ade05000c3413afd1a9faa2ff7c WatchSource:0}: Error finding container 0768b1ca5823ef676103b172c0360ef61b499ade05000c3413afd1a9faa2ff7c: Status 404 returned error can't find the container with id 0768b1ca5823ef676103b172c0360ef61b499ade05000c3413afd1a9faa2ff7c Nov 04 00:30:01.103268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.103177 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp"] Nov 04 00:30:01.108027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.107893 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp"] Nov 04 00:30:01.118454 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:30:01.118429 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbe7e04_0cc5_41fc_ba74_51c8253fc0af.slice/crio-fa32a5e1ddf6789655f557c76305a395b509d2d618f8956a80309bf29fb33411 WatchSource:0}: Error finding container fa32a5e1ddf6789655f557c76305a395b509d2d618f8956a80309bf29fb33411: Status 404 returned error can't find the container with id fa32a5e1ddf6789655f557c76305a395b509d2d618f8956a80309bf29fb33411 Nov 04 00:30:01.388847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.388809 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" event={"ID":"89632be3-5d3a-4b2f-b51c-7f6b21395b5a","Type":"ContainerStarted","Data":"769038ea1112676e8ead4a7d50f272e03b0ccc798e45e44d677371c2502f4b1a"} Nov 04 00:30:01.390212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.390182 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" event={"ID":"49e453cb-0248-4842-936e-c1e2716d822c","Type":"ContainerStarted","Data":"7c4acdb32898d1944202fb28fa480dece646befbbf9fa28eeb466c912b9a15c4"} Nov 04 00:30:01.391384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.391363 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" event={"ID":"9946a8b2-7f05-48c3-9818-4badb91f171e","Type":"ContainerStarted","Data":"fa680dc55c24d33bfec090df99a4034869594434b6e78d05fcb0899aa9c2d2ac"} Nov 04 00:30:01.392584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.392561 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" event={"ID":"3f24a17c-4f5d-461d-a1a1-c08bbd170078","Type":"ContainerStarted","Data":"13d89b2ce029cc98e7782a033977695716af0620bce850939ab0ada8f19d9966"} Nov 04 00:30:01.393872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.393848 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" event={"ID":"4bb1cff7-28c2-40e7-8dda-da12eb866ca6","Type":"ContainerStarted","Data":"0768b1ca5823ef676103b172c0360ef61b499ade05000c3413afd1a9faa2ff7c"} Nov 04 00:30:01.394988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.394968 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" event={"ID":"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af","Type":"ContainerStarted","Data":"fa32a5e1ddf6789655f557c76305a395b509d2d618f8956a80309bf29fb33411"} Nov 04 00:30:01.672450 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:01.672416 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:02.404367 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.404328 6208 generic.go:356] "Generic (PLEG): container finished" podID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerID="200540b0a1ec762fe19ca0db3bb46f2d9ddee004319338d501753a1092b38109" exitCode=60 Nov 04 00:30:02.404855 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.404417 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" event={"ID":"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af","Type":"ContainerDied","Data":"200540b0a1ec762fe19ca0db3bb46f2d9ddee004319338d501753a1092b38109"} Nov 04 00:30:02.404855 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.404682 6208 scope.go:122] "RemoveContainer" containerID="200540b0a1ec762fe19ca0db3bb46f2d9ddee004319338d501753a1092b38109" Nov 04 00:30:02.406267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.406210 6208 generic.go:356] "Generic (PLEG): container finished" podID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerID="c07dd37412bb167020b1eee18f00290373208ad4be5486b106d637acf9a09873" exitCode=60 Nov 04 00:30:02.406377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.406290 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" event={"ID":"89632be3-5d3a-4b2f-b51c-7f6b21395b5a","Type":"ContainerDied","Data":"c07dd37412bb167020b1eee18f00290373208ad4be5486b106d637acf9a09873"} Nov 04 00:30:02.406562 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.406547 6208 scope.go:122] "RemoveContainer" containerID="c07dd37412bb167020b1eee18f00290373208ad4be5486b106d637acf9a09873" Nov 04 00:30:02.409459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.409432 6208 generic.go:356] "Generic (PLEG): container finished" podID="49e453cb-0248-4842-936e-c1e2716d822c" containerID="18703322c404062c53e3167c99debb73d031f2487f811590c8eacc7c8a3d6c6f" exitCode=60 Nov 04 00:30:02.409560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.409530 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" event={"ID":"49e453cb-0248-4842-936e-c1e2716d822c","Type":"ContainerDied","Data":"18703322c404062c53e3167c99debb73d031f2487f811590c8eacc7c8a3d6c6f"} Nov 04 00:30:02.409796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.409782 6208 scope.go:122] "RemoveContainer" containerID="18703322c404062c53e3167c99debb73d031f2487f811590c8eacc7c8a3d6c6f" Nov 04 00:30:02.411352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.411331 6208 generic.go:356] "Generic (PLEG): container finished" podID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerID="1708bfde55d84ed8e415f574a1775a0e80d42a562a7b73362994983b7c7c4c96" exitCode=60 Nov 04 00:30:02.411444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.411400 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" event={"ID":"9946a8b2-7f05-48c3-9818-4badb91f171e","Type":"ContainerDied","Data":"1708bfde55d84ed8e415f574a1775a0e80d42a562a7b73362994983b7c7c4c96"} Nov 04 00:30:02.411641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.411626 6208 scope.go:122] "RemoveContainer" containerID="1708bfde55d84ed8e415f574a1775a0e80d42a562a7b73362994983b7c7c4c96" Nov 04 00:30:02.413490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.413472 6208 generic.go:356] "Generic (PLEG): container finished" podID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerID="cf52bfce19747920cdeb8b817c0d5bb4131158d4df8c7af179c8fcf05d37265f" exitCode=60 Nov 04 00:30:02.413589 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.413559 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" event={"ID":"3f24a17c-4f5d-461d-a1a1-c08bbd170078","Type":"ContainerDied","Data":"cf52bfce19747920cdeb8b817c0d5bb4131158d4df8c7af179c8fcf05d37265f"} Nov 04 00:30:02.413788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.413770 6208 scope.go:122] "RemoveContainer" containerID="cf52bfce19747920cdeb8b817c0d5bb4131158d4df8c7af179c8fcf05d37265f" Nov 04 00:30:02.415970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.415949 6208 generic.go:356] "Generic (PLEG): container finished" podID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerID="95dad8173065dc1f3b6d54683d7744f2b625818c68551394abab8d1f4d251bdd" exitCode=60 Nov 04 00:30:02.416116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.415975 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" event={"ID":"4bb1cff7-28c2-40e7-8dda-da12eb866ca6","Type":"ContainerDied","Data":"95dad8173065dc1f3b6d54683d7744f2b625818c68551394abab8d1f4d251bdd"} Nov 04 00:30:02.416388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.416365 6208 scope.go:122] "RemoveContainer" containerID="95dad8173065dc1f3b6d54683d7744f2b625818c68551394abab8d1f4d251bdd" Nov 04 00:30:02.560640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:02.560604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:03.092639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.092597 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:03.441974 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.441861 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" event={"ID":"89632be3-5d3a-4b2f-b51c-7f6b21395b5a","Type":"ContainerStarted","Data":"b8ed6854cfad2280122f87f37d2161ad277cebb800833152083b44af12e49b6b"} Nov 04 00:30:03.444271 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.444240 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" event={"ID":"49e453cb-0248-4842-936e-c1e2716d822c","Type":"ContainerStarted","Data":"cb443237eae76a7c345063cc0f8a4d23ca0e9b2304db8c7d309fe729b7824a35"} Nov 04 00:30:03.446291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.446259 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" event={"ID":"9946a8b2-7f05-48c3-9818-4badb91f171e","Type":"ContainerStarted","Data":"ec2695b6928e6b874fabda0117e6eca35125aa70b201a5ee2dd78471ecfa2dad"} Nov 04 00:30:03.448594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.448570 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" event={"ID":"3f24a17c-4f5d-461d-a1a1-c08bbd170078","Type":"ContainerStarted","Data":"8cf321bbd13153992595e66fa17e8a5ce58cf5c54d8209ca2000e726c478a6dc"} Nov 04 00:30:03.450470 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.450446 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" event={"ID":"4bb1cff7-28c2-40e7-8dda-da12eb866ca6","Type":"ContainerStarted","Data":"d58316fe03f433347773b60da1e55233d62dd01647bd74db02c356800368caaf"} Nov 04 00:30:03.452236 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.452218 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" event={"ID":"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af","Type":"ContainerStarted","Data":"b0c5e893af2c7134a7c2fb1a26b1b1e46703f2c5fd57d32e1834ffde83a455ee"} Nov 04 00:30:03.467073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.467003 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" podStartSLOduration=3.466989986 podStartE2EDuration="3.466989986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.462739515 +0000 UTC m=+3622.285335676" watchObservedRunningTime="2026-11-04 00:30:03.466989986 +0000 UTC m=+3622.289586148" Nov 04 00:30:03.483546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.483495 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" podStartSLOduration=3.483480667 podStartE2EDuration="3.483480667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.480886562 +0000 UTC m=+3622.303482728" watchObservedRunningTime="2026-11-04 00:30:03.483480667 +0000 UTC m=+3622.306076829" Nov 04 00:30:03.505832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.505766 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" podStartSLOduration=3.5057453880000002 podStartE2EDuration="3.505745388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.504691897 +0000 UTC m=+3622.327288061" watchObservedRunningTime="2026-11-04 00:30:03.505745388 +0000 UTC m=+3622.328341551" Nov 04 00:30:03.566372 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.566322 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" podStartSLOduration=3.566306039 podStartE2EDuration="3.566306039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.562549071 +0000 UTC m=+3622.385145234" watchObservedRunningTime="2026-11-04 00:30:03.566306039 +0000 UTC m=+3622.388902200" Nov 04 00:30:03.584390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.584338 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" podStartSLOduration=3.584322734 podStartE2EDuration="3.584322734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.582972469 +0000 UTC m=+3622.405568632" watchObservedRunningTime="2026-11-04 00:30:03.584322734 +0000 UTC m=+3622.406918896" Nov 04 00:30:03.610608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.610523 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" podStartSLOduration=3.610507923 podStartE2EDuration="3.610507923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:30:03.607432611 +0000 UTC m=+3622.430028773" watchObservedRunningTime="2026-11-04 00:30:03.610507923 +0000 UTC m=+3622.433104097" Nov 04 00:30:03.720615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:03.720581 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:04.462202 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.462168 6208 generic.go:356] "Generic (PLEG): container finished" podID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerID="d58316fe03f433347773b60da1e55233d62dd01647bd74db02c356800368caaf" exitCode=60 Nov 04 00:30:04.462670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.462270 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" event={"ID":"4bb1cff7-28c2-40e7-8dda-da12eb866ca6","Type":"ContainerDied","Data":"d58316fe03f433347773b60da1e55233d62dd01647bd74db02c356800368caaf"} Nov 04 00:30:04.462670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.462340 6208 scope.go:122] "RemoveContainer" containerID="95dad8173065dc1f3b6d54683d7744f2b625818c68551394abab8d1f4d251bdd" Nov 04 00:30:04.462670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.462579 6208 scope.go:122] "RemoveContainer" containerID="d58316fe03f433347773b60da1e55233d62dd01647bd74db02c356800368caaf" Nov 04 00:30:04.463004 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:04.462976 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt_odh-ai-gateway-infra(4bb1cff7-28c2-40e7-8dda-da12eb866ca6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" Nov 04 00:30:04.464357 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.464337 6208 generic.go:356] "Generic (PLEG): container finished" podID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerID="b0c5e893af2c7134a7c2fb1a26b1b1e46703f2c5fd57d32e1834ffde83a455ee" exitCode=60 Nov 04 00:30:04.464465 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.464411 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" event={"ID":"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af","Type":"ContainerDied","Data":"b0c5e893af2c7134a7c2fb1a26b1b1e46703f2c5fd57d32e1834ffde83a455ee"} Nov 04 00:30:04.464752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.464733 6208 scope.go:122] "RemoveContainer" containerID="b0c5e893af2c7134a7c2fb1a26b1b1e46703f2c5fd57d32e1834ffde83a455ee" Nov 04 00:30:04.465100 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:04.465079 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp_odh-ai-gateway-infra(fbbe7e04-0cc5-41fc-ba74-51c8253fc0af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" Nov 04 00:30:04.466389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.466372 6208 generic.go:356] "Generic (PLEG): container finished" podID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerID="b8ed6854cfad2280122f87f37d2161ad277cebb800833152083b44af12e49b6b" exitCode=60 Nov 04 00:30:04.466489 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.466477 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" event={"ID":"89632be3-5d3a-4b2f-b51c-7f6b21395b5a","Type":"ContainerDied","Data":"b8ed6854cfad2280122f87f37d2161ad277cebb800833152083b44af12e49b6b"} Nov 04 00:30:04.466701 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.466683 6208 scope.go:122] "RemoveContainer" containerID="b8ed6854cfad2280122f87f37d2161ad277cebb800833152083b44af12e49b6b" Nov 04 00:30:04.467060 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:04.467005 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws_odh-ai-gateway-infra(89632be3-5d3a-4b2f-b51c-7f6b21395b5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" Nov 04 00:30:04.468564 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.468545 6208 generic.go:356] "Generic (PLEG): container finished" podID="49e453cb-0248-4842-936e-c1e2716d822c" containerID="cb443237eae76a7c345063cc0f8a4d23ca0e9b2304db8c7d309fe729b7824a35" exitCode=60 Nov 04 00:30:04.468660 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.468606 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" event={"ID":"49e453cb-0248-4842-936e-c1e2716d822c","Type":"ContainerDied","Data":"cb443237eae76a7c345063cc0f8a4d23ca0e9b2304db8c7d309fe729b7824a35"} Nov 04 00:30:04.468869 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.468855 6208 scope.go:122] "RemoveContainer" containerID="cb443237eae76a7c345063cc0f8a4d23ca0e9b2304db8c7d309fe729b7824a35" Nov 04 00:30:04.469162 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:04.469143 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95_odh-ai-gateway-infra(49e453cb-0248-4842-936e-c1e2716d822c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" podUID="49e453cb-0248-4842-936e-c1e2716d822c" Nov 04 00:30:04.470648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.470629 6208 generic.go:356] "Generic (PLEG): container finished" podID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerID="ec2695b6928e6b874fabda0117e6eca35125aa70b201a5ee2dd78471ecfa2dad" exitCode=60 Nov 04 00:30:04.470753 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.470688 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" event={"ID":"9946a8b2-7f05-48c3-9818-4badb91f171e","Type":"ContainerDied","Data":"ec2695b6928e6b874fabda0117e6eca35125aa70b201a5ee2dd78471ecfa2dad"} Nov 04 00:30:04.470930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.470915 6208 scope.go:122] "RemoveContainer" containerID="ec2695b6928e6b874fabda0117e6eca35125aa70b201a5ee2dd78471ecfa2dad" Nov 04 00:30:04.471181 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:04.471160 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp_odh-ai-gateway-infra(9946a8b2-7f05-48c3-9818-4badb91f171e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" Nov 04 00:30:04.472678 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.472654 6208 generic.go:356] "Generic (PLEG): container finished" podID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerID="8cf321bbd13153992595e66fa17e8a5ce58cf5c54d8209ca2000e726c478a6dc" exitCode=60 Nov 04 00:30:04.472776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.472717 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" event={"ID":"3f24a17c-4f5d-461d-a1a1-c08bbd170078","Type":"ContainerDied","Data":"8cf321bbd13153992595e66fa17e8a5ce58cf5c54d8209ca2000e726c478a6dc"} Nov 04 00:30:04.472896 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.472882 6208 scope.go:122] "RemoveContainer" containerID="8cf321bbd13153992595e66fa17e8a5ce58cf5c54d8209ca2000e726c478a6dc" Nov 04 00:30:04.473112 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:04.473094 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895870-24sj7_odh-ai-gateway-infra(3f24a17c-4f5d-461d-a1a1-c08bbd170078)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" Nov 04 00:30:04.477192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.477174 6208 scope.go:122] "RemoveContainer" containerID="200540b0a1ec762fe19ca0db3bb46f2d9ddee004319338d501753a1092b38109" Nov 04 00:30:04.492893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.492870 6208 scope.go:122] "RemoveContainer" containerID="c07dd37412bb167020b1eee18f00290373208ad4be5486b106d637acf9a09873" Nov 04 00:30:04.511949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.511835 6208 scope.go:122] "RemoveContainer" containerID="18703322c404062c53e3167c99debb73d031f2487f811590c8eacc7c8a3d6c6f" Nov 04 00:30:04.531325 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.531285 6208 scope.go:122] "RemoveContainer" containerID="1708bfde55d84ed8e415f574a1775a0e80d42a562a7b73362994983b7c7c4c96" Nov 04 00:30:04.545949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.545921 6208 scope.go:122] "RemoveContainer" containerID="cf52bfce19747920cdeb8b817c0d5bb4131158d4df8c7af179c8fcf05d37265f" Nov 04 00:30:04.676929 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.676895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:04.787123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:04.787012 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:05.482191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:05.482164 6208 scope.go:122] "RemoveContainer" containerID="ec2695b6928e6b874fabda0117e6eca35125aa70b201a5ee2dd78471ecfa2dad" Nov 04 00:30:05.482645 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:05.482534 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp_odh-ai-gateway-infra(9946a8b2-7f05-48c3-9818-4badb91f171e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" Nov 04 00:30:05.483824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:05.483802 6208 scope.go:122] "RemoveContainer" containerID="8cf321bbd13153992595e66fa17e8a5ce58cf5c54d8209ca2000e726c478a6dc" Nov 04 00:30:05.484184 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:05.484159 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895870-24sj7_odh-ai-gateway-infra(3f24a17c-4f5d-461d-a1a1-c08bbd170078)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" Nov 04 00:30:05.485667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:05.485648 6208 scope.go:122] "RemoveContainer" containerID="d58316fe03f433347773b60da1e55233d62dd01647bd74db02c356800368caaf" Nov 04 00:30:05.485975 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:05.485954 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt_odh-ai-gateway-infra(4bb1cff7-28c2-40e7-8dda-da12eb866ca6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" Nov 04 00:30:05.487270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:05.487229 6208 scope.go:122] "RemoveContainer" containerID="b0c5e893af2c7134a7c2fb1a26b1b1e46703f2c5fd57d32e1834ffde83a455ee" Nov 04 00:30:05.487568 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:05.487549 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp_odh-ai-gateway-infra(fbbe7e04-0cc5-41fc-ba74-51c8253fc0af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" Nov 04 00:30:05.488702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:05.488683 6208 scope.go:122] "RemoveContainer" containerID="b8ed6854cfad2280122f87f37d2161ad277cebb800833152083b44af12e49b6b" Nov 04 00:30:05.488963 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:05.488941 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws_odh-ai-gateway-infra(89632be3-5d3a-4b2f-b51c-7f6b21395b5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" Nov 04 00:30:05.490280 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:05.490264 6208 scope.go:122] "RemoveContainer" containerID="cb443237eae76a7c345063cc0f8a4d23ca0e9b2304db8c7d309fe729b7824a35" Nov 04 00:30:05.490486 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:05.490469 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95_odh-ai-gateway-infra(49e453cb-0248-4842-936e-c1e2716d822c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" podUID="49e453cb-0248-4842-936e-c1e2716d822c" Nov 04 00:30:05.707398 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:05.707358 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:30:05.837638 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:05.837536 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:06.946953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:06.946917 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:07.062101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:07.062062 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:07.732155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:07.732114 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:08.609235 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:08.609196 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:09.431236 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:09.431199 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:10.213608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:10.213570 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:10.958853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:10.958816 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:12.006884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:12.005752 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:12.866878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:12.866838 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:13.752220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:13.752190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:14.287529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:14.287496 6208 scope.go:122] "RemoveContainer" containerID="b8ed6854cfad2280122f87f37d2161ad277cebb800833152083b44af12e49b6b" Nov 04 00:30:14.536916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:14.536881 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:14.804163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:14.804130 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:15.288006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:15.287973 6208 scope.go:122] "RemoveContainer" containerID="cb443237eae76a7c345063cc0f8a4d23ca0e9b2304db8c7d309fe729b7824a35" Nov 04 00:30:15.354324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:15.354287 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:15.574797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:15.574763 6208 generic.go:356] "Generic (PLEG): container finished" podID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerID="08f81c73427d063e9b80e5b205b0e5b55bc494f05a34740c6b2c0819c5e616bc" exitCode=60 Nov 04 00:30:15.575012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:15.574828 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" event={"ID":"89632be3-5d3a-4b2f-b51c-7f6b21395b5a","Type":"ContainerDied","Data":"08f81c73427d063e9b80e5b205b0e5b55bc494f05a34740c6b2c0819c5e616bc"} Nov 04 00:30:15.575012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:15.574886 6208 scope.go:122] "RemoveContainer" containerID="b8ed6854cfad2280122f87f37d2161ad277cebb800833152083b44af12e49b6b" Nov 04 00:30:15.575499 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:15.575375 6208 scope.go:122] "RemoveContainer" containerID="08f81c73427d063e9b80e5b205b0e5b55bc494f05a34740c6b2c0819c5e616bc" Nov 04 00:30:15.575835 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:15.575811 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws_odh-ai-gateway-infra(89632be3-5d3a-4b2f-b51c-7f6b21395b5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" Nov 04 00:30:16.152499 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.152457 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:16.287660 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.287622 6208 scope.go:122] "RemoveContainer" containerID="b0c5e893af2c7134a7c2fb1a26b1b1e46703f2c5fd57d32e1834ffde83a455ee" Nov 04 00:30:16.287867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.287796 6208 scope.go:122] "RemoveContainer" containerID="d58316fe03f433347773b60da1e55233d62dd01647bd74db02c356800368caaf" Nov 04 00:30:16.585600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.585568 6208 generic.go:356] "Generic (PLEG): container finished" podID="49e453cb-0248-4842-936e-c1e2716d822c" containerID="1e7d2de66d0be70867651a16871ed17358a82c4777870669ddc47097af461ac2" exitCode=60 Nov 04 00:30:16.585738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.585633 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" event={"ID":"49e453cb-0248-4842-936e-c1e2716d822c","Type":"ContainerDied","Data":"1e7d2de66d0be70867651a16871ed17358a82c4777870669ddc47097af461ac2"} Nov 04 00:30:16.585738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.585693 6208 scope.go:122] "RemoveContainer" containerID="cb443237eae76a7c345063cc0f8a4d23ca0e9b2304db8c7d309fe729b7824a35" Nov 04 00:30:16.586136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.586113 6208 scope.go:122] "RemoveContainer" containerID="1e7d2de66d0be70867651a16871ed17358a82c4777870669ddc47097af461ac2" Nov 04 00:30:16.586546 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:16.586521 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95_odh-ai-gateway-infra(49e453cb-0248-4842-936e-c1e2716d822c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" podUID="49e453cb-0248-4842-936e-c1e2716d822c" Nov 04 00:30:16.609270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.609235 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws"] Nov 04 00:30:16.784091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.784066 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:16.945623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.945527 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-kube-api-access-l9c8m\" (UniqueName: \"kubernetes.io/projected/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-kube-api-access-l9c8m\") pod \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\" (UID: \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\") " Nov 04 00:30:16.945787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.945713 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-maas-api-service-ca\") pod \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\" (UID: \"89632be3-5d3a-4b2f-b51c-7f6b21395b5a\") " Nov 04 00:30:16.946128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.946092 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-maas-api-service-ca" pod "89632be3-5d3a-4b2f-b51c-7f6b21395b5a" (UID: "89632be3-5d3a-4b2f-b51c-7f6b21395b5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:30:16.946520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.946501 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:16.947761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:16.947731 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-kube-api-access-l9c8m" pod "89632be3-5d3a-4b2f-b51c-7f6b21395b5a" (UID: "89632be3-5d3a-4b2f-b51c-7f6b21395b5a"). InnerVolumeSpecName "kube-api-access-l9c8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:30:17.047979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.047938 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9c8m\" (UniqueName: \"kubernetes.io/projected/89632be3-5d3a-4b2f-b51c-7f6b21395b5a-kube-api-access-l9c8m\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:17.124359 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.124302 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:17.287558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.287518 6208 scope.go:122] "RemoveContainer" containerID="8cf321bbd13153992595e66fa17e8a5ce58cf5c54d8209ca2000e726c478a6dc" Nov 04 00:30:17.287969 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.287566 6208 scope.go:122] "RemoveContainer" containerID="ec2695b6928e6b874fabda0117e6eca35125aa70b201a5ee2dd78471ecfa2dad" Nov 04 00:30:17.597728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.597691 6208 generic.go:356] "Generic (PLEG): container finished" podID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerID="08f0ae1fde764e1e7ed270a195492673823f77af3fb7c3a770b7bafdda4da7a1" exitCode=60 Nov 04 00:30:17.597883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.597748 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" event={"ID":"4bb1cff7-28c2-40e7-8dda-da12eb866ca6","Type":"ContainerDied","Data":"08f0ae1fde764e1e7ed270a195492673823f77af3fb7c3a770b7bafdda4da7a1"} Nov 04 00:30:17.598078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.597845 6208 scope.go:122] "RemoveContainer" containerID="d58316fe03f433347773b60da1e55233d62dd01647bd74db02c356800368caaf" Nov 04 00:30:17.598432 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.598398 6208 scope.go:122] "RemoveContainer" containerID="08f0ae1fde764e1e7ed270a195492673823f77af3fb7c3a770b7bafdda4da7a1" Nov 04 00:30:17.598870 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:17.598839 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt_odh-ai-gateway-infra(4bb1cff7-28c2-40e7-8dda-da12eb866ca6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" Nov 04 00:30:17.600461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.600434 6208 generic.go:356] "Generic (PLEG): container finished" podID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerID="766dbf541598bd9382eb7e6f3473d47ec4d09954a6e2be4ae435781a2c97e225" exitCode=60 Nov 04 00:30:17.600551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.600528 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" event={"ID":"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af","Type":"ContainerDied","Data":"766dbf541598bd9382eb7e6f3473d47ec4d09954a6e2be4ae435781a2c97e225"} Nov 04 00:30:17.601701 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.601682 6208 scope.go:122] "RemoveContainer" containerID="766dbf541598bd9382eb7e6f3473d47ec4d09954a6e2be4ae435781a2c97e225" Nov 04 00:30:17.602217 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:17.602190 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp_odh-ai-gateway-infra(fbbe7e04-0cc5-41fc-ba74-51c8253fc0af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" Nov 04 00:30:17.602344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.602215 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" event={"ID":"89632be3-5d3a-4b2f-b51c-7f6b21395b5a","Type":"ContainerDied","Data":"769038ea1112676e8ead4a7d50f272e03b0ccc798e45e44d677371c2502f4b1a"} Nov 04 00:30:17.602344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.602279 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws" Nov 04 00:30:17.612800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.612772 6208 scope.go:122] "RemoveContainer" containerID="b0c5e893af2c7134a7c2fb1a26b1b1e46703f2c5fd57d32e1834ffde83a455ee" Nov 04 00:30:17.620734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.620703 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95"] Nov 04 00:30:17.636027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.636003 6208 scope.go:122] "RemoveContainer" containerID="08f81c73427d063e9b80e5b205b0e5b55bc494f05a34740c6b2c0819c5e616bc" Nov 04 00:30:17.665249 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.665214 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws"] Nov 04 00:30:17.671670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.671636 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2697a0-29895870-frgws"] Nov 04 00:30:17.813071 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.811422 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:17.862485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.862370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:17.962172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.962137 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e453cb-0248-4842-936e-c1e2716d822c-kube-api-access-7skrj\" (UniqueName: \"kubernetes.io/projected/49e453cb-0248-4842-936e-c1e2716d822c-kube-api-access-7skrj\") pod \"49e453cb-0248-4842-936e-c1e2716d822c\" (UID: \"49e453cb-0248-4842-936e-c1e2716d822c\") " Nov 04 00:30:17.962349 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.962310 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e453cb-0248-4842-936e-c1e2716d822c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e453cb-0248-4842-936e-c1e2716d822c-maas-api-service-ca\") pod \"49e453cb-0248-4842-936e-c1e2716d822c\" (UID: \"49e453cb-0248-4842-936e-c1e2716d822c\") " Nov 04 00:30:17.962811 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.962773 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e453cb-0248-4842-936e-c1e2716d822c-maas-api-service-ca" pod "49e453cb-0248-4842-936e-c1e2716d822c" (UID: "49e453cb-0248-4842-936e-c1e2716d822c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:30:17.964627 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:17.964597 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e453cb-0248-4842-936e-c1e2716d822c-kube-api-access-7skrj" pod "49e453cb-0248-4842-936e-c1e2716d822c" (UID: "49e453cb-0248-4842-936e-c1e2716d822c"). InnerVolumeSpecName "kube-api-access-7skrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:30:18.064428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.064383 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e453cb-0248-4842-936e-c1e2716d822c-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:18.064428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.064420 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7skrj\" (UniqueName: \"kubernetes.io/projected/49e453cb-0248-4842-936e-c1e2716d822c-kube-api-access-7skrj\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:18.610670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.610587 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" Nov 04 00:30:18.611159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.610592 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95" event={"ID":"49e453cb-0248-4842-936e-c1e2716d822c","Type":"ContainerDied","Data":"7c4acdb32898d1944202fb28fa480dece646befbbf9fa28eeb466c912b9a15c4"} Nov 04 00:30:18.611159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.610739 6208 scope.go:122] "RemoveContainer" containerID="1e7d2de66d0be70867651a16871ed17358a82c4777870669ddc47097af461ac2" Nov 04 00:30:18.612726 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.612701 6208 generic.go:356] "Generic (PLEG): container finished" podID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerID="dc20891131e5355799e4e62c0d6a36a6898a6c959a49efb2c9beaf56c6dcd1cc" exitCode=60 Nov 04 00:30:18.612866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.612780 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" event={"ID":"9946a8b2-7f05-48c3-9818-4badb91f171e","Type":"ContainerDied","Data":"dc20891131e5355799e4e62c0d6a36a6898a6c959a49efb2c9beaf56c6dcd1cc"} Nov 04 00:30:18.613351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.613327 6208 scope.go:122] "RemoveContainer" containerID="dc20891131e5355799e4e62c0d6a36a6898a6c959a49efb2c9beaf56c6dcd1cc" Nov 04 00:30:18.613737 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:18.613713 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp_odh-ai-gateway-infra(9946a8b2-7f05-48c3-9818-4badb91f171e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" Nov 04 00:30:18.615292 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.615265 6208 generic.go:356] "Generic (PLEG): container finished" podID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerID="242fdad44e126f744b6972bbe7af135fac8b06608bdd2f7e82da42d36a8b4805" exitCode=60 Nov 04 00:30:18.615405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.615341 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" event={"ID":"3f24a17c-4f5d-461d-a1a1-c08bbd170078","Type":"ContainerDied","Data":"242fdad44e126f744b6972bbe7af135fac8b06608bdd2f7e82da42d36a8b4805"} Nov 04 00:30:18.615736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.615716 6208 scope.go:122] "RemoveContainer" containerID="242fdad44e126f744b6972bbe7af135fac8b06608bdd2f7e82da42d36a8b4805" Nov 04 00:30:18.616021 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:18.615998 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895870-24sj7_odh-ai-gateway-infra(3f24a17c-4f5d-461d-a1a1-c08bbd170078)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" Nov 04 00:30:18.633491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.633462 6208 scope.go:122] "RemoveContainer" containerID="ec2695b6928e6b874fabda0117e6eca35125aa70b201a5ee2dd78471ecfa2dad" Nov 04 00:30:18.638355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.638327 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt"] Nov 04 00:30:18.652405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.652312 6208 scope.go:122] "RemoveContainer" containerID="8cf321bbd13153992595e66fa17e8a5ce58cf5c54d8209ca2000e726c478a6dc" Nov 04 00:30:18.664055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.655651 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp"] Nov 04 00:30:18.687054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.687008 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95"] Nov 04 00:30:18.696463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.696425 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2341ca-29895870-4pk95"] Nov 04 00:30:18.783748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.783712 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:18.955076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:18.955042 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:19.016850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.016821 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:19.083978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.083945 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-maas-api-service-ca\") pod \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\" (UID: \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\") " Nov 04 00:30:19.084174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.084127 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-kube-api-access-drtpp\" (UniqueName: \"kubernetes.io/projected/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-kube-api-access-drtpp\") pod \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\" (UID: \"4bb1cff7-28c2-40e7-8dda-da12eb866ca6\") " Nov 04 00:30:19.084406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.084382 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-maas-api-service-ca" pod "4bb1cff7-28c2-40e7-8dda-da12eb866ca6" (UID: "4bb1cff7-28c2-40e7-8dda-da12eb866ca6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:30:19.084636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.084616 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:19.086171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.086149 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-kube-api-access-drtpp" pod "4bb1cff7-28c2-40e7-8dda-da12eb866ca6" (UID: "4bb1cff7-28c2-40e7-8dda-da12eb866ca6"). InnerVolumeSpecName "kube-api-access-drtpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:30:19.185821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.185751 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-kube-api-access-56fms\" (UniqueName: \"kubernetes.io/projected/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-kube-api-access-56fms\") pod \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\" (UID: \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\") " Nov 04 00:30:19.185966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.185890 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-maas-api-service-ca\") pod \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\" (UID: \"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af\") " Nov 04 00:30:19.186281 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.186256 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-maas-api-service-ca" pod "fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" (UID: "fbbe7e04-0cc5-41fc-ba74-51c8253fc0af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:30:19.186507 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.186490 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:19.186568 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.186514 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drtpp\" (UniqueName: \"kubernetes.io/projected/4bb1cff7-28c2-40e7-8dda-da12eb866ca6-kube-api-access-drtpp\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:19.187788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.187766 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-kube-api-access-56fms" pod "fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" (UID: "fbbe7e04-0cc5-41fc-ba74-51c8253fc0af"). InnerVolumeSpecName "kube-api-access-56fms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:30:19.288424 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.288385 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fms\" (UniqueName: \"kubernetes.io/projected/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af-kube-api-access-56fms\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:19.297046 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.297010 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e453cb-0248-4842-936e-c1e2716d822c" path="/var/lib/kubelet/pods/49e453cb-0248-4842-936e-c1e2716d822c/volumes" Nov 04 00:30:19.297515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.297500 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" path="/var/lib/kubelet/pods/89632be3-5d3a-4b2f-b51c-7f6b21395b5a/volumes" Nov 04 00:30:19.609345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.609308 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:19.635020 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.634993 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" Nov 04 00:30:19.635020 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.635003 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt" event={"ID":"4bb1cff7-28c2-40e7-8dda-da12eb866ca6","Type":"ContainerDied","Data":"0768b1ca5823ef676103b172c0360ef61b499ade05000c3413afd1a9faa2ff7c"} Nov 04 00:30:19.635584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.635111 6208 scope.go:122] "RemoveContainer" containerID="08f0ae1fde764e1e7ed270a195492673823f77af3fb7c3a770b7bafdda4da7a1" Nov 04 00:30:19.636515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.636495 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" Nov 04 00:30:19.636635 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.636542 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp" event={"ID":"fbbe7e04-0cc5-41fc-ba74-51c8253fc0af","Type":"ContainerDied","Data":"fa32a5e1ddf6789655f557c76305a395b509d2d618f8956a80309bf29fb33411"} Nov 04 00:30:19.646524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.646507 6208 scope.go:122] "RemoveContainer" containerID="766dbf541598bd9382eb7e6f3473d47ec4d09954a6e2be4ae435781a2c97e225" Nov 04 00:30:19.652476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.652423 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp"] Nov 04 00:30:19.665463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.665440 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp"] Nov 04 00:30:19.671515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.671492 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-330eb5-29895870-7sbdp"] Nov 04 00:30:19.679313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.679290 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7"] Nov 04 00:30:19.695481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.695448 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt"] Nov 04 00:30:19.706408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.706373 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1b5fca-29895870-74gvt"] Nov 04 00:30:19.721485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.721453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:19.974073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:19.971230 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:20.051189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.051156 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:20.112160 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.112108 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9946a8b2-7f05-48c3-9818-4badb91f171e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946a8b2-7f05-48c3-9818-4badb91f171e-maas-api-service-ca\") pod \"9946a8b2-7f05-48c3-9818-4badb91f171e\" (UID: \"9946a8b2-7f05-48c3-9818-4badb91f171e\") " Nov 04 00:30:20.112411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.112258 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9946a8b2-7f05-48c3-9818-4badb91f171e-kube-api-access-msznv\" (UniqueName: \"kubernetes.io/projected/9946a8b2-7f05-48c3-9818-4badb91f171e-kube-api-access-msznv\") pod \"9946a8b2-7f05-48c3-9818-4badb91f171e\" (UID: \"9946a8b2-7f05-48c3-9818-4badb91f171e\") " Nov 04 00:30:20.112568 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.112544 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9946a8b2-7f05-48c3-9818-4badb91f171e-maas-api-service-ca" pod "9946a8b2-7f05-48c3-9818-4badb91f171e" (UID: "9946a8b2-7f05-48c3-9818-4badb91f171e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:30:20.113015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.112982 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9946a8b2-7f05-48c3-9818-4badb91f171e-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:20.114688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.114665 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9946a8b2-7f05-48c3-9818-4badb91f171e-kube-api-access-msznv" pod "9946a8b2-7f05-48c3-9818-4badb91f171e" (UID: "9946a8b2-7f05-48c3-9818-4badb91f171e"). InnerVolumeSpecName "kube-api-access-msznv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:30:20.213681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.213652 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f24a17c-4f5d-461d-a1a1-c08bbd170078-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a17c-4f5d-461d-a1a1-c08bbd170078-maas-api-service-ca\") pod \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\" (UID: \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\") " Nov 04 00:30:20.213853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.213821 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f24a17c-4f5d-461d-a1a1-c08bbd170078-kube-api-access-krh6c\" (UniqueName: \"kubernetes.io/projected/3f24a17c-4f5d-461d-a1a1-c08bbd170078-kube-api-access-krh6c\") pod \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\" (UID: \"3f24a17c-4f5d-461d-a1a1-c08bbd170078\") " Nov 04 00:30:20.214006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.213982 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f24a17c-4f5d-461d-a1a1-c08bbd170078-maas-api-service-ca" pod "3f24a17c-4f5d-461d-a1a1-c08bbd170078" (UID: "3f24a17c-4f5d-461d-a1a1-c08bbd170078"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:30:20.214426 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.214408 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a17c-4f5d-461d-a1a1-c08bbd170078-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:20.214498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.214428 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msznv\" (UniqueName: \"kubernetes.io/projected/9946a8b2-7f05-48c3-9818-4badb91f171e-kube-api-access-msznv\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:20.215961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.215938 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f24a17c-4f5d-461d-a1a1-c08bbd170078-kube-api-access-krh6c" pod "3f24a17c-4f5d-461d-a1a1-c08bbd170078" (UID: "3f24a17c-4f5d-461d-a1a1-c08bbd170078"). InnerVolumeSpecName "kube-api-access-krh6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:30:20.315830 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.315786 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krh6c\" (UniqueName: \"kubernetes.io/projected/3f24a17c-4f5d-461d-a1a1-c08bbd170078-kube-api-access-krh6c\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:20.650256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.650169 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" Nov 04 00:30:20.650256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.650193 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp" event={"ID":"9946a8b2-7f05-48c3-9818-4badb91f171e","Type":"ContainerDied","Data":"fa680dc55c24d33bfec090df99a4034869594434b6e78d05fcb0899aa9c2d2ac"} Nov 04 00:30:20.650792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.650258 6208 scope.go:122] "RemoveContainer" containerID="dc20891131e5355799e4e62c0d6a36a6898a6c959a49efb2c9beaf56c6dcd1cc" Nov 04 00:30:20.652050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.652019 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" Nov 04 00:30:20.652177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.652059 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7" event={"ID":"3f24a17c-4f5d-461d-a1a1-c08bbd170078","Type":"ContainerDied","Data":"13d89b2ce029cc98e7782a033977695716af0620bce850939ab0ada8f19d9966"} Nov 04 00:30:20.664532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.664512 6208 scope.go:122] "RemoveContainer" containerID="242fdad44e126f744b6972bbe7af135fac8b06608bdd2f7e82da42d36a8b4805" Nov 04 00:30:20.682512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.682487 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7"] Nov 04 00:30:20.688657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.688633 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895870-24sj7"] Nov 04 00:30:20.704730 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.704704 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp"] Nov 04 00:30:20.715891 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.715864 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f6f292-29895870-b59pp"] Nov 04 00:30:20.927831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:20.927747 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:21.298360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:21.298327 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" path="/var/lib/kubelet/pods/3f24a17c-4f5d-461d-a1a1-c08bbd170078/volumes" Nov 04 00:30:21.298777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:21.298763 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" path="/var/lib/kubelet/pods/4bb1cff7-28c2-40e7-8dda-da12eb866ca6/volumes" Nov 04 00:30:21.299181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:21.299168 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" path="/var/lib/kubelet/pods/9946a8b2-7f05-48c3-9818-4badb91f171e/volumes" Nov 04 00:30:21.299865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:21.299850 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" path="/var/lib/kubelet/pods/fbbe7e04-0cc5-41fc-ba74-51c8253fc0af/volumes" Nov 04 00:30:21.954053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:21.953865 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:22.977295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:22.977258 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:23.905365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:23.905331 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:24.478524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:24.478482 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:25.431394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:25.431360 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:26.229577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:26.229540 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:27.053216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:27.053179 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:28.114348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:28.114307 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:28.917640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:28.917598 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:29.835724 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:29.835691 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:30.655576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:30.655531 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:31.637403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:31.637368 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:32.528477 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:32.528436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:32.838343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:32.838251 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:33.538598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:33.538554 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:33.646574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:33.646534 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:34.312896 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:34.312854 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:35.023094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:35.023054 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:35.331655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:35.331547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:35.434372 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:35.434336 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:35.711527 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:35.711494 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:30:36.012025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:36.011924 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:36.123070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:36.123015 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:36.463093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:36.457517 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:36.938117 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:36.938002 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:37.039287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:37.039250 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:37.147072 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:37.147020 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:37.252534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:37.252498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:37.940612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:37.940563 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:38.760103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:38.760070 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:39.644667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:39.644625 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:40.416534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:40.416491 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:41.225195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:41.225161 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:42.075984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:42.075943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:43.037612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:43.037580 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:43.585062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:43.585000 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:44.166932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:44.166895 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:44.883269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:44.883226 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:44.982914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:44.982882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:45.364426 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:45.364396 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:45.640879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:45.640795 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:46.466425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:46.466383 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:47.244761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:47.244718 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:48.027682 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:48.027646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:48.730611 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:48.730563 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:49.455489 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:49.455450 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:50.228985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:50.228945 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:51.226748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:51.226709 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:51.433056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:51.426425 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:51.527612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:51.527473 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:52.256343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:52.256309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:52.620501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:52.620391 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:53.049955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:53.049918 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:53.846190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:53.846156 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:54.631135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:54.631097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:55.341491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:55.341456 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:56.054308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:56.054271 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:56.855955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:56.855918 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:57.562831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:57.562794 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c"] Nov 04 00:30:57.563097 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:57.563067 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" podUID="79986d29-e237-4bbc-bb59-9687d182ca4c" containerName="maas-api" containerID="cri-o://3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be" gracePeriod=30 Nov 04 00:30:57.682584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:57.682552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:57.851509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:57.851389 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:58.007375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.007346 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:30:58.016711 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.016681 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:58.020216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.020193 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-metrics-certs\") pod \"79986d29-e237-4bbc-bb59-9687d182ca4c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " Nov 04 00:30:58.020313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.020262 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-maas-api-tls\") pod \"79986d29-e237-4bbc-bb59-9687d182ca4c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " Nov 04 00:30:58.020381 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.020341 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79986d29-e237-4bbc-bb59-9687d182ca4c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79986d29-e237-4bbc-bb59-9687d182ca4c-openshift-service-ca\") pod \"79986d29-e237-4bbc-bb59-9687d182ca4c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " Nov 04 00:30:58.020449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.020383 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79986d29-e237-4bbc-bb59-9687d182ca4c-kube-api-access-j27kp\" (UniqueName: \"kubernetes.io/projected/79986d29-e237-4bbc-bb59-9687d182ca4c-kube-api-access-j27kp\") pod \"79986d29-e237-4bbc-bb59-9687d182ca4c\" (UID: \"79986d29-e237-4bbc-bb59-9687d182ca4c\") " Nov 04 00:30:58.020759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.020729 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79986d29-e237-4bbc-bb59-9687d182ca4c-openshift-service-ca" pod "79986d29-e237-4bbc-bb59-9687d182ca4c" (UID: "79986d29-e237-4bbc-bb59-9687d182ca4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:30:58.020929 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.020912 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79986d29-e237-4bbc-bb59-9687d182ca4c-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:58.022338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.022306 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-metrics-certs" pod "79986d29-e237-4bbc-bb59-9687d182ca4c" (UID: "79986d29-e237-4bbc-bb59-9687d182ca4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:30:58.022338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.022324 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-maas-api-tls" pod "79986d29-e237-4bbc-bb59-9687d182ca4c" (UID: "79986d29-e237-4bbc-bb59-9687d182ca4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:30:58.022483 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.022464 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79986d29-e237-4bbc-bb59-9687d182ca4c-kube-api-access-j27kp" pod "79986d29-e237-4bbc-bb59-9687d182ca4c" (UID: "79986d29-e237-4bbc-bb59-9687d182ca4c"). InnerVolumeSpecName "kube-api-access-j27kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:30:58.116376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.116274 6208 generic.go:356] "Generic (PLEG): container finished" podID="79986d29-e237-4bbc-bb59-9687d182ca4c" containerID="3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be" exitCode=0 Nov 04 00:30:58.116376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.116333 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" event={"ID":"79986d29-e237-4bbc-bb59-9687d182ca4c","Type":"ContainerDied","Data":"3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be"} Nov 04 00:30:58.116376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.116373 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" event={"ID":"79986d29-e237-4bbc-bb59-9687d182ca4c","Type":"ContainerDied","Data":"c7e4e0d0fc5f76cd600730faf42feef1cd74e6957c54d453fd6e65b5d7ab340c"} Nov 04 00:30:58.116697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.116394 6208 scope.go:122] "RemoveContainer" containerID="3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be" Nov 04 00:30:58.116697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.116514 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c" Nov 04 00:30:58.124694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.124663 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:58.124803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.124699 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79986d29-e237-4bbc-bb59-9687d182ca4c-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:58.124803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.124715 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j27kp\" (UniqueName: \"kubernetes.io/projected/79986d29-e237-4bbc-bb59-9687d182ca4c-kube-api-access-j27kp\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:30:58.140158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.136160 6208 scope.go:122] "RemoveContainer" containerID="3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be" Nov 04 00:30:58.140158 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:30:58.139900 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be\": container with ID starting with 3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be not found: ID does not exist" containerID="3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be" Nov 04 00:30:58.140158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.139941 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be"} err="failed to get container status \"3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be\": rpc error: code = NotFound desc = could not find container \"3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be\": container with ID starting with 3d05dbb67276f24b10a4fa54e66659148edefe496599e325f52c6f14030b71be not found: ID does not exist" Nov 04 00:30:58.160116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.160057 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c"] Nov 04 00:30:58.165704 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:58.165676 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3a002b-75cf489d5f-87c8c"] Nov 04 00:30:59.013075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:59.005955 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:59.215667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:59.215620 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:30:59.295981 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:59.295900 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79986d29-e237-4bbc-bb59-9687d182ca4c" path="/var/lib/kubelet/pods/79986d29-e237-4bbc-bb59-9687d182ca4c/volumes" Nov 04 00:30:59.745900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:30:59.745867 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:00.314916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:00.314877 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:00.411280 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:00.411243 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:00.552466 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:00.552431 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:00.821807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:00.821768 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:01.225926 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:01.225897 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:01.345425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:01.345377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:01.824724 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:01.824682 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:02.036601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:02.036570 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:02.721353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:02.721317 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:03.618746 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:03.618707 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:04.359914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:04.359878 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:05.100311 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:05.100274 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:05.706056 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:31:05.705999 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:31:05.916057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:05.907019 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:06.840800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:06.840766 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:07.830705 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:07.828601 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:08.791642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:08.791600 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:09.627819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:09.627781 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:10.070310 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:10.070278 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:10.167560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:10.167525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:10.359378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:10.359278 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:10.683028 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:10.682930 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:11.528192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:11.528142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:12.100930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.100896 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8"] Nov 04 00:31:12.101243 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.101213 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" podUID="64445549-ad95-4060-9522-ab6b550cecbf" containerName="maas-api" containerID="cri-o://4c0e1021d5aede26f1ca6d88d379d72274e897e3e99309e9759f6fca3d36d7fb" gracePeriod=30 Nov 04 00:31:12.134573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.134547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:12.484108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.463306 6208 generic.go:356] "Generic (PLEG): container finished" podID="64445549-ad95-4060-9522-ab6b550cecbf" containerID="4c0e1021d5aede26f1ca6d88d379d72274e897e3e99309e9759f6fca3d36d7fb" exitCode=0 Nov 04 00:31:12.484108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.463553 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" event={"ID":"64445549-ad95-4060-9522-ab6b550cecbf","Type":"ContainerDied","Data":"4c0e1021d5aede26f1ca6d88d379d72274e897e3e99309e9759f6fca3d36d7fb"} Nov 04 00:31:12.509052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.507503 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:12.600992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.600962 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:31:12.678553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.678518 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64445549-ad95-4060-9522-ab6b550cecbf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64445549-ad95-4060-9522-ab6b550cecbf-openshift-service-ca\") pod \"64445549-ad95-4060-9522-ab6b550cecbf\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " Nov 04 00:31:12.678747 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.678571 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64445549-ad95-4060-9522-ab6b550cecbf-kube-api-access-ppndz\" (UniqueName: \"kubernetes.io/projected/64445549-ad95-4060-9522-ab6b550cecbf-kube-api-access-ppndz\") pod \"64445549-ad95-4060-9522-ab6b550cecbf\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " Nov 04 00:31:12.678747 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.678650 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-metrics-certs\") pod \"64445549-ad95-4060-9522-ab6b550cecbf\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " Nov 04 00:31:12.678747 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.678714 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls\") pod \"64445549-ad95-4060-9522-ab6b550cecbf\" (UID: \"64445549-ad95-4060-9522-ab6b550cecbf\") " Nov 04 00:31:12.678970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.678943 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64445549-ad95-4060-9522-ab6b550cecbf-openshift-service-ca" pod "64445549-ad95-4060-9522-ab6b550cecbf" (UID: "64445549-ad95-4060-9522-ab6b550cecbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:31:12.679297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.679271 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64445549-ad95-4060-9522-ab6b550cecbf-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:12.681442 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.681416 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-metrics-certs" pod "64445549-ad95-4060-9522-ab6b550cecbf" (UID: "64445549-ad95-4060-9522-ab6b550cecbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:31:12.681578 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.681561 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64445549-ad95-4060-9522-ab6b550cecbf-kube-api-access-ppndz" pod "64445549-ad95-4060-9522-ab6b550cecbf" (UID: "64445549-ad95-4060-9522-ab6b550cecbf"). InnerVolumeSpecName "kube-api-access-ppndz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:31:12.681799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.681784 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls" pod "64445549-ad95-4060-9522-ab6b550cecbf" (UID: "64445549-ad95-4060-9522-ab6b550cecbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:31:12.781366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.781286 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppndz\" (UniqueName: \"kubernetes.io/projected/64445549-ad95-4060-9522-ab6b550cecbf-kube-api-access-ppndz\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:12.781366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.781314 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:12.781366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:12.781328 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64445549-ad95-4060-9522-ab6b550cecbf-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:13.242129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:13.242086 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:13.472436 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:13.472403 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" Nov 04 00:31:13.472636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:13.472431 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8" event={"ID":"64445549-ad95-4060-9522-ab6b550cecbf","Type":"ContainerDied","Data":"e1585b9636c7866b7dce82792b54fb243f39b4362563c8938da0a3132061d812"} Nov 04 00:31:13.472636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:13.472504 6208 scope.go:122] "RemoveContainer" containerID="4c0e1021d5aede26f1ca6d88d379d72274e897e3e99309e9759f6fca3d36d7fb" Nov 04 00:31:13.492373 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:13.492322 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8"] Nov 04 00:31:13.495698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:13.495677 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1b5fca-6fcc789bb4-8btf8"] Nov 04 00:31:13.634909 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:13.634870 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:14.230647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:14.230606 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:14.946209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:14.946170 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:15.296494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:15.296460 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64445549-ad95-4060-9522-ab6b550cecbf" path="/var/lib/kubelet/pods/64445549-ad95-4060-9522-ab6b550cecbf/volumes" Nov 04 00:31:15.729054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:15.728721 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:16.584059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:16.580749 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:17.334761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:17.334704 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:17.723887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:17.723857 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:17.829470 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:17.829436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:18.051421 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:18.051332 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:18.279659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:18.279626 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:18.596459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:18.596424 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55"] Nov 04 00:31:18.596660 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:18.596637 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" containerID="cri-o://55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70" gracePeriod=30 Nov 04 00:31:18.631366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:18.631176 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:19.113665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:19.113627 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:19.221576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:19.221540 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:19.444677 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:19.444592 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:19.541367 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:19.541338 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:19.941731 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:19.941691 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:20.742752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:20.742714 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:21.545309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:21.545264 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:22.395308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:22.395256 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:23.152088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:23.152045 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:23.739958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:23.739925 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 04 00:31:23.740377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:23.739995 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 04 00:31:23.902649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:23.902624 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.026239 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5979d19-2211-458a-9734-0a9a8ddfd617-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5979d19-2211-458a-9734-0a9a8ddfd617-istiod-ca-cert\") pod \"f5979d19-2211-458a-9734-0a9a8ddfd617\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.026343 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-podinfo\") pod \"f5979d19-2211-458a-9734-0a9a8ddfd617\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.026384 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-certs\") pod \"f5979d19-2211-458a-9734-0a9a8ddfd617\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.026504 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-envoy\") pod \"f5979d19-2211-458a-9734-0a9a8ddfd617\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.026539 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-credential-socket\") pod \"f5979d19-2211-458a-9734-0a9a8ddfd617\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.026576 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-socket\") pod \"f5979d19-2211-458a-9734-0a9a8ddfd617\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.026638 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-data\") pod \"f5979d19-2211-458a-9734-0a9a8ddfd617\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.026785 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-token\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-token\") pod \"f5979d19-2211-458a-9734-0a9a8ddfd617\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.026882 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-kube-api-access-2ljzq\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-kube-api-access-2ljzq\") pod \"f5979d19-2211-458a-9734-0a9a8ddfd617\" (UID: \"f5979d19-2211-458a-9734-0a9a8ddfd617\") " Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.027710 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-socket" pod "f5979d19-2211-458a-9734-0a9a8ddfd617" (UID: "f5979d19-2211-458a-9734-0a9a8ddfd617"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.027960 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-credential-socket" pod "f5979d19-2211-458a-9734-0a9a8ddfd617" (UID: "f5979d19-2211-458a-9734-0a9a8ddfd617"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.029093 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-data" pod "f5979d19-2211-458a-9734-0a9a8ddfd617" (UID: "f5979d19-2211-458a-9734-0a9a8ddfd617"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.029934 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.029956 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:24.034131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.029969 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:24.035771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.035431 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5979d19-2211-458a-9734-0a9a8ddfd617-istiod-ca-cert" pod "f5979d19-2211-458a-9734-0a9a8ddfd617" (UID: "f5979d19-2211-458a-9734-0a9a8ddfd617"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:31:24.037146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.037004 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-certs" pod "f5979d19-2211-458a-9734-0a9a8ddfd617" (UID: "f5979d19-2211-458a-9734-0a9a8ddfd617"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:31:24.106636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.106579 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-kube-api-access-2ljzq" pod "f5979d19-2211-458a-9734-0a9a8ddfd617" (UID: "f5979d19-2211-458a-9734-0a9a8ddfd617"). InnerVolumeSpecName "kube-api-access-2ljzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:31:24.106958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.106928 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-token" pod "f5979d19-2211-458a-9734-0a9a8ddfd617" (UID: "f5979d19-2211-458a-9734-0a9a8ddfd617"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:31:24.118267 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.118203 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-podinfo" pod "f5979d19-2211-458a-9734-0a9a8ddfd617" (UID: "f5979d19-2211-458a-9734-0a9a8ddfd617"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:31:24.120246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.118812 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-envoy" pod "f5979d19-2211-458a-9734-0a9a8ddfd617" (UID: "f5979d19-2211-458a-9734-0a9a8ddfd617"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:31:24.130979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.130947 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5979d19-2211-458a-9734-0a9a8ddfd617-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:24.130979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.130981 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:24.134259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.130996 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:24.134259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.131008 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:24.134259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.131022 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:24.134259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.131061 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ljzq\" (UniqueName: \"kubernetes.io/projected/f5979d19-2211-458a-9734-0a9a8ddfd617-kube-api-access-2ljzq\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:24.136274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.136175 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:24.553100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.553063 6208 generic.go:356] "Generic (PLEG): container finished" podID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerID="55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70" exitCode=0 Nov 04 00:31:24.553100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.553099 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" Nov 04 00:31:24.553389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.553147 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" event={"ID":"f5979d19-2211-458a-9734-0a9a8ddfd617","Type":"ContainerDied","Data":"55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70"} Nov 04 00:31:24.553389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.553207 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55" event={"ID":"f5979d19-2211-458a-9734-0a9a8ddfd617","Type":"ContainerDied","Data":"34d7906197471247e08acabe00a532919d94509e20d16ec966967ac8b2a9fb4e"} Nov 04 00:31:24.553389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.553235 6208 scope.go:122] "RemoveContainer" containerID="55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70" Nov 04 00:31:24.579558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.579535 6208 scope.go:122] "RemoveContainer" containerID="eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df" Nov 04 00:31:24.588482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.588451 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55"] Nov 04 00:31:24.594197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.594163 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-1b5fca-openshift-default-75b565c489-dfz55"] Nov 04 00:31:24.596708 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.596687 6208 scope.go:122] "RemoveContainer" containerID="55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70" Nov 04 00:31:24.597066 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:31:24.597024 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70\": container with ID starting with 55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70 not found: ID does not exist" containerID="55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70" Nov 04 00:31:24.597135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.597080 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70"} err="failed to get container status \"55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70\": rpc error: code = NotFound desc = could not find container \"55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70\": container with ID starting with 55001804e2efa8ac98808de5e533becb773a77a096c8d32c790b913382f2be70 not found: ID does not exist" Nov 04 00:31:24.597135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.597105 6208 scope.go:122] "RemoveContainer" containerID="eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df" Nov 04 00:31:24.597421 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:31:24.597393 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df\": container with ID starting with eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df not found: ID does not exist" containerID="eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df" Nov 04 00:31:24.597476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:24.597433 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df"} err="failed to get container status \"eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df\": rpc error: code = NotFound desc = could not find container \"eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df\": container with ID starting with eaa42deacc4771e313d7bd5d2d025297338526ffe8af0f9ec21695fb57d469df not found: ID does not exist" Nov 04 00:31:25.035254 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:25.035213 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:25.297386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:25.297301 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" path="/var/lib/kubelet/pods/f5979d19-2211-458a-9734-0a9a8ddfd617/volumes" Nov 04 00:31:25.836144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:25.836106 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:26.477387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:26.477348 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:26.572416 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:26.572389 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:26.962176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:26.962106 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:27.061204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:27.061169 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:27.788338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:27.788295 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:28.060388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:28.060268 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:28.304813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:28.304779 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:28.878647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:28.878614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:29.741181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:29.741143 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:30.243965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:30.243927 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:30.961536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:30.961498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:32.055337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:32.055298 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:32.854265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:32.854231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:33.416050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:33.416006 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:33.748868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:33.748828 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x"] Nov 04 00:31:33.755273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:33.755246 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt"] Nov 04 00:31:33.819023 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:33.818989 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:33.923677 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:33.923641 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:34.036109 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:34.027567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:35.006265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:35.006231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:35.705219 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:31:35.705180 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:31:35.796755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:35.796719 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:35.926908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:35.926871 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:36.770379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:36.770343 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:37.524698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:37.524661 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:38.364380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:38.364346 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:38.749663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:38.749621 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" podUID="cabc3bde-6478-4a87-af40-513329416597" containerName="payload-processing" containerID="cri-o://5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a" gracePeriod=25 Nov 04 00:31:38.756098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:38.756068 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" podUID="d2fe6082-38e3-4e74-8712-4a1e9977cfaa" containerName="payload-pre-processing" containerID="cri-o://e9aa6728475ec95d3e4142fa68c72c7a79c338e11ff6604a9162cff9948c5c8a" gracePeriod=25 Nov 04 00:31:39.026707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.026680 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:31:39.097783 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.097746 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabc3bde-6478-4a87-af40-513329416597-kube-api-access-cwn7s\" (UniqueName: \"kubernetes.io/projected/cabc3bde-6478-4a87-af40-513329416597-kube-api-access-cwn7s\") pod \"cabc3bde-6478-4a87-af40-513329416597\" (UID: \"cabc3bde-6478-4a87-af40-513329416597\") " Nov 04 00:31:39.097970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.097936 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabc3bde-6478-4a87-af40-513329416597-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc3bde-6478-4a87-af40-513329416597-plugins-config-volume\") pod \"cabc3bde-6478-4a87-af40-513329416597\" (UID: \"cabc3bde-6478-4a87-af40-513329416597\") " Nov 04 00:31:39.098399 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.098371 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabc3bde-6478-4a87-af40-513329416597-plugins-config-volume" pod "cabc3bde-6478-4a87-af40-513329416597" (UID: "cabc3bde-6478-4a87-af40-513329416597"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:31:39.098697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.098680 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cabc3bde-6478-4a87-af40-513329416597-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:39.099883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.099858 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabc3bde-6478-4a87-af40-513329416597-kube-api-access-cwn7s" pod "cabc3bde-6478-4a87-af40-513329416597" (UID: "cabc3bde-6478-4a87-af40-513329416597"). InnerVolumeSpecName "kube-api-access-cwn7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:31:39.200084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.200028 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwn7s\" (UniqueName: \"kubernetes.io/projected/cabc3bde-6478-4a87-af40-513329416597-kube-api-access-cwn7s\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:39.322531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.322499 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:39.667055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.666945 6208 generic.go:356] "Generic (PLEG): container finished" podID="cabc3bde-6478-4a87-af40-513329416597" containerID="5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a" exitCode=0 Nov 04 00:31:39.667055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.667002 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" Nov 04 00:31:39.667583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.667028 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" event={"ID":"cabc3bde-6478-4a87-af40-513329416597","Type":"ContainerDied","Data":"5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a"} Nov 04 00:31:39.667583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.667367 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x" event={"ID":"cabc3bde-6478-4a87-af40-513329416597","Type":"ContainerDied","Data":"1205141cb1c52501936752dab10905d8646ba82f1d7f29dba2eed2b3c5c9421b"} Nov 04 00:31:39.667583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.667406 6208 scope.go:122] "RemoveContainer" containerID="5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a" Nov 04 00:31:39.681187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.681170 6208 scope.go:122] "RemoveContainer" containerID="5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a" Nov 04 00:31:39.681444 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:31:39.681422 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a\": container with ID starting with 5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a not found: ID does not exist" containerID="5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a" Nov 04 00:31:39.681511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.681449 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a"} err="failed to get container status \"5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a\": rpc error: code = NotFound desc = could not find container \"5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a\": container with ID starting with 5e9f8685fe2516aa91153f19c47f48ad5d9ef86f3cd8d2472238035a6a6d0e9a not found: ID does not exist" Nov 04 00:31:39.688080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.688053 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x"] Nov 04 00:31:39.691876 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:39.691853 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-71b2f3-f9bd7cc86-27b5x"] Nov 04 00:31:40.038114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:40.038079 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:40.823609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:40.823575 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:41.296610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:41.296578 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabc3bde-6478-4a87-af40-513329416597" path="/var/lib/kubelet/pods/cabc3bde-6478-4a87-af40-513329416597/volumes" Nov 04 00:31:41.617641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:41.617560 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:42.308517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:42.308486 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:42.753749 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:42.753719 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:43.374956 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:43.374924 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:43.475577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:43.475543 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:43.642332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:43.642232 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:44.233817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:44.233770 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:44.764344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:44.764306 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" podUID="d2fe6082-38e3-4e74-8712-4a1e9977cfaa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:31:44.932580 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:44.932533 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:45.653389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:45.653353 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:46.437512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:46.437476 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:47.324738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:47.324698 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:48.053419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:48.053372 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:48.977595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:48.977561 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:49.930850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:49.930801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:50.758360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:50.758329 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:51.668587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:51.668552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:52.413633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:52.413598 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:52.524435 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:52.524360 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:52.930211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:52.930174 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:53.222095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:53.222024 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:53.327108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:53.327074 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:53.760713 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:53.760679 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:54.112453 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:54.112366 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:54.615429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:54.615398 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:54.764460 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:54.764426 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" podUID="d2fe6082-38e3-4e74-8712-4a1e9977cfaa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:31:55.454847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:55.454806 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:56.244955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:56.244924 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:57.028305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:57.028263 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:57.675734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:57.675694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:57.827226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:57.827193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:58.648670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:58.648633 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:58.943940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:58.943609 6208 generic.go:356] "Generic (PLEG): container finished" podID="d2fe6082-38e3-4e74-8712-4a1e9977cfaa" containerID="e9aa6728475ec95d3e4142fa68c72c7a79c338e11ff6604a9162cff9948c5c8a" exitCode=0 Nov 04 00:31:58.943940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:58.943871 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" event={"ID":"d2fe6082-38e3-4e74-8712-4a1e9977cfaa","Type":"ContainerDied","Data":"e9aa6728475ec95d3e4142fa68c72c7a79c338e11ff6604a9162cff9948c5c8a"} Nov 04 00:31:59.040548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.040523 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:31:59.126131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.126089 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-plugins-config-volume\") pod \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\" (UID: \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\") " Nov 04 00:31:59.126314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.126213 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-kube-api-access-c6c84\" (UniqueName: \"kubernetes.io/projected/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-kube-api-access-c6c84\") pod \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\" (UID: \"d2fe6082-38e3-4e74-8712-4a1e9977cfaa\") " Nov 04 00:31:59.126556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.126528 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-plugins-config-volume" pod "d2fe6082-38e3-4e74-8712-4a1e9977cfaa" (UID: "d2fe6082-38e3-4e74-8712-4a1e9977cfaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:31:59.126776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.126757 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:59.128322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.128298 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-kube-api-access-c6c84" pod "d2fe6082-38e3-4e74-8712-4a1e9977cfaa" (UID: "d2fe6082-38e3-4e74-8712-4a1e9977cfaa"). InnerVolumeSpecName "kube-api-access-c6c84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:31:59.227898 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.227867 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6c84\" (UniqueName: \"kubernetes.io/projected/d2fe6082-38e3-4e74-8712-4a1e9977cfaa-kube-api-access-c6c84\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:31:59.293122 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.292896 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:59.390181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.390148 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:59.495611 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.495525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:31:59.952319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.952243 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" event={"ID":"d2fe6082-38e3-4e74-8712-4a1e9977cfaa","Type":"ContainerDied","Data":"9dc946e902427d6a00f552edbbed25e0032dd0420bdba5456422c4d3e488c3a1"} Nov 04 00:31:59.952319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.952279 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt" Nov 04 00:31:59.952319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.952300 6208 scope.go:122] "RemoveContainer" containerID="e9aa6728475ec95d3e4142fa68c72c7a79c338e11ff6604a9162cff9948c5c8a" Nov 04 00:31:59.972352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.972327 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt"] Nov 04 00:31:59.978627 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:31:59.978603 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-71b2f3-8bfdd769f-j57gt"] Nov 04 00:32:00.238179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:00.238146 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:00.940681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:00.940644 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:01.296767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:01.296731 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fe6082-38e3-4e74-8712-4a1e9977cfaa" path="/var/lib/kubelet/pods/d2fe6082-38e3-4e74-8712-4a1e9977cfaa/volumes" Nov 04 00:32:01.748352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:01.748312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:01.848841 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:01.848810 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:02.438120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:02.434847 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:02.542438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:02.542401 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:02.713139 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:02.713105 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:02.977732 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:02.977638 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:03.460494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:03.460460 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:03.720835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:03.720724 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:04.470608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:04.470572 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:05.459233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:05.459200 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:05.704785 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:32:05.704745 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:32:06.221550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:06.221511 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:06.919902 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:06.919871 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:07.723675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:07.723637 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:08.515233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:08.515194 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:09.250190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:09.250145 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:10.151152 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:10.151106 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:10.455431 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:10.455399 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:10.554928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:10.554887 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:10.768158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:10.768071 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:11.076674 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:11.076578 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:11.176866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:11.176834 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:11.810758 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:11.810722 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:12.654856 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:12.654818 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:13.268305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:13.268258 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:13.660865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:13.660777 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:14.433397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:14.433360 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:15.148314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:15.148274 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:15.715333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:15.715301 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9"] Nov 04 00:32:15.721914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:15.718323 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" podUID="e53bfbe1-15dc-4b22-8121-5e927de1be27" containerName="maas-api" containerID="cri-o://4ea8f6d1c5c0dc114f19b652b6c58432ee1a7384749fef75c9913eac205bece3" gracePeriod=30 Nov 04 00:32:15.814759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:15.814730 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:16.100410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.100314 6208 generic.go:356] "Generic (PLEG): container finished" podID="e53bfbe1-15dc-4b22-8121-5e927de1be27" containerID="4ea8f6d1c5c0dc114f19b652b6c58432ee1a7384749fef75c9913eac205bece3" exitCode=0 Nov 04 00:32:16.100594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.100491 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" event={"ID":"e53bfbe1-15dc-4b22-8121-5e927de1be27","Type":"ContainerDied","Data":"4ea8f6d1c5c0dc114f19b652b6c58432ee1a7384749fef75c9913eac205bece3"} Nov 04 00:32:16.142802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.142775 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:32:16.167565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.167523 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:16.170147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.170120 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-metrics-certs\") pod \"e53bfbe1-15dc-4b22-8121-5e927de1be27\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " Nov 04 00:32:16.170275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.170200 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53bfbe1-15dc-4b22-8121-5e927de1be27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53bfbe1-15dc-4b22-8121-5e927de1be27-openshift-service-ca\") pod \"e53bfbe1-15dc-4b22-8121-5e927de1be27\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " Nov 04 00:32:16.170410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.170395 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-maas-api-tls\") pod \"e53bfbe1-15dc-4b22-8121-5e927de1be27\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " Nov 04 00:32:16.170489 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.170477 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53bfbe1-15dc-4b22-8121-5e927de1be27-kube-api-access-kbxn7\" (UniqueName: \"kubernetes.io/projected/e53bfbe1-15dc-4b22-8121-5e927de1be27-kube-api-access-kbxn7\") pod \"e53bfbe1-15dc-4b22-8121-5e927de1be27\" (UID: \"e53bfbe1-15dc-4b22-8121-5e927de1be27\") " Nov 04 00:32:16.170653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.170612 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53bfbe1-15dc-4b22-8121-5e927de1be27-openshift-service-ca" pod "e53bfbe1-15dc-4b22-8121-5e927de1be27" (UID: "e53bfbe1-15dc-4b22-8121-5e927de1be27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:32:16.171190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.171166 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53bfbe1-15dc-4b22-8121-5e927de1be27-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:16.173217 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.173186 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-maas-api-tls" pod "e53bfbe1-15dc-4b22-8121-5e927de1be27" (UID: "e53bfbe1-15dc-4b22-8121-5e927de1be27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:32:16.173316 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.173238 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53bfbe1-15dc-4b22-8121-5e927de1be27-kube-api-access-kbxn7" pod "e53bfbe1-15dc-4b22-8121-5e927de1be27" (UID: "e53bfbe1-15dc-4b22-8121-5e927de1be27"). InnerVolumeSpecName "kube-api-access-kbxn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:16.175837 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.175811 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-metrics-certs" pod "e53bfbe1-15dc-4b22-8121-5e927de1be27" (UID: "e53bfbe1-15dc-4b22-8121-5e927de1be27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:32:16.272504 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.272466 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:16.272656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.272512 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbxn7\" (UniqueName: \"kubernetes.io/projected/e53bfbe1-15dc-4b22-8121-5e927de1be27-kube-api-access-kbxn7\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:16.272656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.272529 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53bfbe1-15dc-4b22-8121-5e927de1be27-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:16.941820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:16.941780 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:17.109832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:17.109800 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" Nov 04 00:32:17.109832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:17.109822 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9" event={"ID":"e53bfbe1-15dc-4b22-8121-5e927de1be27","Type":"ContainerDied","Data":"851983b749c4f2d96277359a5d1321a3470c42a5bebbf7be307adc80eb31d8c2"} Nov 04 00:32:17.110127 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:17.109886 6208 scope.go:122] "RemoveContainer" containerID="4ea8f6d1c5c0dc114f19b652b6c58432ee1a7384749fef75c9913eac205bece3" Nov 04 00:32:17.139481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:17.139436 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9"] Nov 04 00:32:17.143642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:17.143618 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7d218-b9cdcc697-gxrh9"] Nov 04 00:32:17.296285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:17.296249 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53bfbe1-15dc-4b22-8121-5e927de1be27" path="/var/lib/kubelet/pods/e53bfbe1-15dc-4b22-8121-5e927de1be27/volumes" Nov 04 00:32:17.629828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:17.629748 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:17.843752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:17.843711 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:17.943505 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:17.943424 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:18.160602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:18.160571 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:18.514928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:18.514894 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:18.640056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:18.629913 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:19.006759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:19.006720 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:19.107237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:19.107194 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:19.379006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:19.378926 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l"] Nov 04 00:32:19.379215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:19.379183 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" containerID="cri-o://a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754" gracePeriod=30 Nov 04 00:32:19.409789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:19.409761 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:20.037784 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:20.037753 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:20.200157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:20.200119 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:20.692903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:20.692861 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:32:20.693094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:20.692941 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:32:20.770432 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:20.770398 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:21.525341 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:21.525300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:22.307482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:22.307445 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:23.165142 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:23.165081 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:24.064142 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.064104 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:24.091027 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.090990 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp"] Nov 04 00:32:24.219162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.219111 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:24.650940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.650882 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:32:24.665128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665099 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-socket\") pod \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " Nov 04 00:32:24.665247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665142 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-envoy\") pod \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " Nov 04 00:32:24.665247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665173 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-podinfo\") pod \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " Nov 04 00:32:24.665247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665193 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-kube-api-access-j8rdv\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-kube-api-access-j8rdv\") pod \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " Nov 04 00:32:24.665397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665360 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istiod-ca-cert\") pod \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " Nov 04 00:32:24.665471 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665393 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-socket" pod "b1457a0f-48fe-45d4-a180-ae1d84f549ba" (UID: "b1457a0f-48fe-45d4-a180-ae1d84f549ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:24.665545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665530 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-certs\") pod \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " Nov 04 00:32:24.665608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665580 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-credential-socket\") pod \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " Nov 04 00:32:24.665709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665694 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-data\") pod \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " Nov 04 00:32:24.665766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665736 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-token\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-token\") pod \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\" (UID: \"b1457a0f-48fe-45d4-a180-ae1d84f549ba\") " Nov 04 00:32:24.665881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665856 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-certs" pod "b1457a0f-48fe-45d4-a180-ae1d84f549ba" (UID: "b1457a0f-48fe-45d4-a180-ae1d84f549ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:24.665992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.665966 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-credential-socket" pod "b1457a0f-48fe-45d4-a180-ae1d84f549ba" (UID: "b1457a0f-48fe-45d4-a180-ae1d84f549ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:24.666178 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.666108 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-data" pod "b1457a0f-48fe-45d4-a180-ae1d84f549ba" (UID: "b1457a0f-48fe-45d4-a180-ae1d84f549ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:24.666333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.666307 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istiod-ca-cert" pod "b1457a0f-48fe-45d4-a180-ae1d84f549ba" (UID: "b1457a0f-48fe-45d4-a180-ae1d84f549ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:32:24.666509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.666487 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:24.666570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.666514 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:24.666570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.666531 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:24.666570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.666544 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:24.667661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.667641 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-podinfo" pod "b1457a0f-48fe-45d4-a180-ae1d84f549ba" (UID: "b1457a0f-48fe-45d4-a180-ae1d84f549ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:32:24.667925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.667905 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-envoy" pod "b1457a0f-48fe-45d4-a180-ae1d84f549ba" (UID: "b1457a0f-48fe-45d4-a180-ae1d84f549ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:24.668017 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.667967 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-kube-api-access-j8rdv" pod "b1457a0f-48fe-45d4-a180-ae1d84f549ba" (UID: "b1457a0f-48fe-45d4-a180-ae1d84f549ba"). InnerVolumeSpecName "kube-api-access-j8rdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:24.668144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.668023 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-token" pod "b1457a0f-48fe-45d4-a180-ae1d84f549ba" (UID: "b1457a0f-48fe-45d4-a180-ae1d84f549ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:24.752497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.752459 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:24.767749 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.767717 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:24.767749 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.767748 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:24.767955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.767758 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rdv\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-kube-api-access-j8rdv\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:24.767955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.767767 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:24.767955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.767777 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1457a0f-48fe-45d4-a180-ae1d84f549ba-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:24.937802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:24.937717 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:25.195291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.195203 6208 generic.go:356] "Generic (PLEG): container finished" podID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerID="a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754" exitCode=0 Nov 04 00:32:25.195291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.195267 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" Nov 04 00:32:25.195291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.195277 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" event={"ID":"b1457a0f-48fe-45d4-a180-ae1d84f549ba","Type":"ContainerDied","Data":"a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754"} Nov 04 00:32:25.195553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.195337 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l" event={"ID":"b1457a0f-48fe-45d4-a180-ae1d84f549ba","Type":"ContainerDied","Data":"d71062344d42651046cbf22a03eaf78e3af966ba7da0f30b233e68c84e8acb4b"} Nov 04 00:32:25.195553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.195358 6208 scope.go:122] "RemoveContainer" containerID="a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754" Nov 04 00:32:25.207228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.207209 6208 scope.go:122] "RemoveContainer" containerID="407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2" Nov 04 00:32:25.217933 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.217914 6208 scope.go:122] "RemoveContainer" containerID="a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754" Nov 04 00:32:25.218311 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:32:25.218286 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754\": container with ID starting with a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754 not found: ID does not exist" containerID="a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754" Nov 04 00:32:25.218463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.218319 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754"} err="failed to get container status \"a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754\": rpc error: code = NotFound desc = could not find container \"a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754\": container with ID starting with a76d9c5c5617ba25032748109ae8b7f46ba4c0e4c1c419ce6d038be149ca0754 not found: ID does not exist" Nov 04 00:32:25.218463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.218341 6208 scope.go:122] "RemoveContainer" containerID="407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2" Nov 04 00:32:25.218641 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:32:25.218623 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2\": container with ID starting with 407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2 not found: ID does not exist" containerID="407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2" Nov 04 00:32:25.218698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.218652 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2"} err="failed to get container status \"407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2\": rpc error: code = NotFound desc = could not find container \"407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2\": container with ID starting with 407ecca4c7ead0f1eb32cba78bf203c0c6b8a7697ef55816e36d9764cefc13d2 not found: ID does not exist" Nov 04 00:32:25.221246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.221223 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l"] Nov 04 00:32:25.224613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.224593 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7d218-openshift-default-65c775d56f-4k89l"] Nov 04 00:32:25.296231 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.296198 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" path="/var/lib/kubelet/pods/b1457a0f-48fe-45d4-a180-ae1d84f549ba/volumes" Nov 04 00:32:25.729332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:25.729296 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:26.292925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:26.292893 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:26.392581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:26.392542 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:26.817449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:26.817414 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:27.705776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:27.705742 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:28.008664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:28.008570 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:28.388522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:28.388437 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94"] Nov 04 00:32:28.388670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:28.388636 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" containerID="cri-o://89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e" gracePeriod=30 Nov 04 00:32:28.557175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:28.557137 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:29.414837 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:29.414801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:30.314253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:30.314208 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:31.122663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:31.122629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:31.362900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:31.362860 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t"] Nov 04 00:32:31.382214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:31.382123 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9"] Nov 04 00:32:31.428467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:31.428429 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:31.529184 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:31.529138 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:32.208202 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:32.208165 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:32.388943 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:32.388904 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:33.063668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.063637 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:33.706559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.706529 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 04 00:32:33.738531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.738500 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-socket\") pod \"a31de50e-34be-4e03-8c70-e98d3d4cb577\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " Nov 04 00:32:33.738676 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.738543 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-kube-api-access-vnm5s\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-kube-api-access-vnm5s\") pod \"a31de50e-34be-4e03-8c70-e98d3d4cb577\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " Nov 04 00:32:33.738676 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.738621 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-data\") pod \"a31de50e-34be-4e03-8c70-e98d3d4cb577\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " Nov 04 00:32:33.738801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.738675 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-envoy\") pod \"a31de50e-34be-4e03-8c70-e98d3d4cb577\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " Nov 04 00:32:33.738801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.738704 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-podinfo\") pod \"a31de50e-34be-4e03-8c70-e98d3d4cb577\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " Nov 04 00:32:33.738801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.738790 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-certs\") pod \"a31de50e-34be-4e03-8c70-e98d3d4cb577\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " Nov 04 00:32:33.738945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.738827 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-token\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-token\") pod \"a31de50e-34be-4e03-8c70-e98d3d4cb577\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " Nov 04 00:32:33.738945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.738865 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31de50e-34be-4e03-8c70-e98d3d4cb577-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31de50e-34be-4e03-8c70-e98d3d4cb577-istiod-ca-cert\") pod \"a31de50e-34be-4e03-8c70-e98d3d4cb577\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " Nov 04 00:32:33.738945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.738906 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-credential-socket\") pod \"a31de50e-34be-4e03-8c70-e98d3d4cb577\" (UID: \"a31de50e-34be-4e03-8c70-e98d3d4cb577\") " Nov 04 00:32:33.739548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.739522 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-credential-socket" pod "a31de50e-34be-4e03-8c70-e98d3d4cb577" (UID: "a31de50e-34be-4e03-8c70-e98d3d4cb577"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:33.739728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.739711 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-socket" pod "a31de50e-34be-4e03-8c70-e98d3d4cb577" (UID: "a31de50e-34be-4e03-8c70-e98d3d4cb577"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:33.741372 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.741343 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-data" pod "a31de50e-34be-4e03-8c70-e98d3d4cb577" (UID: "a31de50e-34be-4e03-8c70-e98d3d4cb577"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:33.741600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.741581 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31de50e-34be-4e03-8c70-e98d3d4cb577-istiod-ca-cert" pod "a31de50e-34be-4e03-8c70-e98d3d4cb577" (UID: "a31de50e-34be-4e03-8c70-e98d3d4cb577"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:32:33.742175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.742146 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-certs" pod "a31de50e-34be-4e03-8c70-e98d3d4cb577" (UID: "a31de50e-34be-4e03-8c70-e98d3d4cb577"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:33.743893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.743863 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-envoy" pod "a31de50e-34be-4e03-8c70-e98d3d4cb577" (UID: "a31de50e-34be-4e03-8c70-e98d3d4cb577"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:33.744623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.744603 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-kube-api-access-vnm5s" pod "a31de50e-34be-4e03-8c70-e98d3d4cb577" (UID: "a31de50e-34be-4e03-8c70-e98d3d4cb577"). InnerVolumeSpecName "kube-api-access-vnm5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:33.767789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.767761 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-podinfo" pod "a31de50e-34be-4e03-8c70-e98d3d4cb577" (UID: "a31de50e-34be-4e03-8c70-e98d3d4cb577"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:32:33.768760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.768727 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-token" pod "a31de50e-34be-4e03-8c70-e98d3d4cb577" (UID: "a31de50e-34be-4e03-8c70-e98d3d4cb577"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:33.835088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.835025 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:33.840276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.840243 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:33.840276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.840271 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:33.840446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.840282 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnm5s\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-kube-api-access-vnm5s\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:33.840446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.840293 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:33.840446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.840302 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:33.840446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.840310 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:33.840446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.840319 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a31de50e-34be-4e03-8c70-e98d3d4cb577-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:33.840446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.840328 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a31de50e-34be-4e03-8c70-e98d3d4cb577-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:33.840446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:33.840336 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a31de50e-34be-4e03-8c70-e98d3d4cb577-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:34.262480 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.262445 6208 generic.go:356] "Generic (PLEG): container finished" podID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerID="89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e" exitCode=0 Nov 04 00:32:34.262664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.262498 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" Nov 04 00:32:34.262664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.262528 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" event={"ID":"a31de50e-34be-4e03-8c70-e98d3d4cb577","Type":"ContainerDied","Data":"89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e"} Nov 04 00:32:34.262664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.262609 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94" event={"ID":"a31de50e-34be-4e03-8c70-e98d3d4cb577","Type":"ContainerDied","Data":"2ad43171f96518ebabb4a6f2aa93eac3e8fa3e6c15458925d65fe590af4a7b54"} Nov 04 00:32:34.262664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.262633 6208 scope.go:122] "RemoveContainer" containerID="89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e" Nov 04 00:32:34.273995 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.273978 6208 scope.go:122] "RemoveContainer" containerID="89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e" Nov 04 00:32:34.274297 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:32:34.274275 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e\": container with ID starting with 89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e not found: ID does not exist" containerID="89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e" Nov 04 00:32:34.274361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.274310 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e"} err="failed to get container status \"89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e\": rpc error: code = NotFound desc = could not find container \"89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e\": container with ID starting with 89bdf0cbef2d747516e0a3b4711a3f940d9c737971d84153c48758eb3329d09e not found: ID does not exist" Nov 04 00:32:34.286942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.286907 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94"] Nov 04 00:32:34.291233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.291209 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5cb349-openshift-default-657d95f7f-hgm94"] Nov 04 00:32:34.646659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:34.646546 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:35.295908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:35.295873 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" path="/var/lib/kubelet/pods/a31de50e-34be-4e03-8c70-e98d3d4cb577/volumes" Nov 04 00:32:35.365904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:35.365872 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:35.465850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:35.465815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:35.705912 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:32:35.705878 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:32:36.071355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.071268 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66"] Nov 04 00:32:36.071597 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.071566 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" podUID="5b675c79-e14d-440f-b32b-24dd6bd05427" containerName="maas-api" containerID="cri-o://665111bb2f9b03e52d6529d54b6eb4401052ae8d0e20f5573c1726dfcec4fa64" gracePeriod=30 Nov 04 00:32:36.114879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.114845 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:36.371111 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.368116 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" podUID="127f6328-65eb-4bea-898b-f01e05d6a6dc" containerName="payload-processing" containerID="cri-o://8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6" gracePeriod=25 Nov 04 00:32:36.391613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.391404 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" podUID="b62d2e16-32d3-46b2-8795-4fed31a1c458" containerName="payload-pre-processing" containerID="cri-o://4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1" gracePeriod=25 Nov 04 00:32:36.431701 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.431593 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:36.437077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.436887 6208 generic.go:356] "Generic (PLEG): container finished" podID="5b675c79-e14d-440f-b32b-24dd6bd05427" containerID="665111bb2f9b03e52d6529d54b6eb4401052ae8d0e20f5573c1726dfcec4fa64" exitCode=0 Nov 04 00:32:36.437408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.437316 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" event={"ID":"5b675c79-e14d-440f-b32b-24dd6bd05427","Type":"ContainerDied","Data":"665111bb2f9b03e52d6529d54b6eb4401052ae8d0e20f5573c1726dfcec4fa64"} Nov 04 00:32:36.582824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.582801 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 04 00:32:36.697212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.697177 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-maas-api-tls\") pod \"5b675c79-e14d-440f-b32b-24dd6bd05427\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " Nov 04 00:32:36.697405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.697354 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b675c79-e14d-440f-b32b-24dd6bd05427-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b675c79-e14d-440f-b32b-24dd6bd05427-openshift-service-ca\") pod \"5b675c79-e14d-440f-b32b-24dd6bd05427\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " Nov 04 00:32:36.697470 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.697425 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b675c79-e14d-440f-b32b-24dd6bd05427-kube-api-access-vsnjm\" (UniqueName: \"kubernetes.io/projected/5b675c79-e14d-440f-b32b-24dd6bd05427-kube-api-access-vsnjm\") pod \"5b675c79-e14d-440f-b32b-24dd6bd05427\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " Nov 04 00:32:36.697529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.697510 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-metrics-certs\") pod \"5b675c79-e14d-440f-b32b-24dd6bd05427\" (UID: \"5b675c79-e14d-440f-b32b-24dd6bd05427\") " Nov 04 00:32:36.698277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.698012 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b675c79-e14d-440f-b32b-24dd6bd05427-openshift-service-ca" pod "5b675c79-e14d-440f-b32b-24dd6bd05427" (UID: "5b675c79-e14d-440f-b32b-24dd6bd05427"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:32:36.698277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.698248 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b675c79-e14d-440f-b32b-24dd6bd05427-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:36.700963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.700924 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b675c79-e14d-440f-b32b-24dd6bd05427-kube-api-access-vsnjm" pod "5b675c79-e14d-440f-b32b-24dd6bd05427" (UID: "5b675c79-e14d-440f-b32b-24dd6bd05427"). InnerVolumeSpecName "kube-api-access-vsnjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:36.701148 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.701120 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-maas-api-tls" pod "5b675c79-e14d-440f-b32b-24dd6bd05427" (UID: "5b675c79-e14d-440f-b32b-24dd6bd05427"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:32:36.704388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.704357 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-metrics-certs" pod "5b675c79-e14d-440f-b32b-24dd6bd05427" (UID: "5b675c79-e14d-440f-b32b-24dd6bd05427"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:32:36.756558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.756528 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 04 00:32:36.799018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.798842 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127f6328-65eb-4bea-898b-f01e05d6a6dc-kube-api-access-mmfmg\" (UniqueName: \"kubernetes.io/projected/127f6328-65eb-4bea-898b-f01e05d6a6dc-kube-api-access-mmfmg\") pod \"127f6328-65eb-4bea-898b-f01e05d6a6dc\" (UID: \"127f6328-65eb-4bea-898b-f01e05d6a6dc\") " Nov 04 00:32:36.799197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.799072 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127f6328-65eb-4bea-898b-f01e05d6a6dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127f6328-65eb-4bea-898b-f01e05d6a6dc-plugins-config-volume\") pod \"127f6328-65eb-4bea-898b-f01e05d6a6dc\" (UID: \"127f6328-65eb-4bea-898b-f01e05d6a6dc\") " Nov 04 00:32:36.799647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.799622 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsnjm\" (UniqueName: \"kubernetes.io/projected/5b675c79-e14d-440f-b32b-24dd6bd05427-kube-api-access-vsnjm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:36.799737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.799652 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:36.799737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.799667 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b675c79-e14d-440f-b32b-24dd6bd05427-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:36.800095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.800057 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127f6328-65eb-4bea-898b-f01e05d6a6dc-plugins-config-volume" pod "127f6328-65eb-4bea-898b-f01e05d6a6dc" (UID: "127f6328-65eb-4bea-898b-f01e05d6a6dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:32:36.814330 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.814291 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127f6328-65eb-4bea-898b-f01e05d6a6dc-kube-api-access-mmfmg" pod "127f6328-65eb-4bea-898b-f01e05d6a6dc" (UID: "127f6328-65eb-4bea-898b-f01e05d6a6dc"). InnerVolumeSpecName "kube-api-access-mmfmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:36.815482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.815457 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 04 00:32:36.900861 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.900766 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62d2e16-32d3-46b2-8795-4fed31a1c458-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62d2e16-32d3-46b2-8795-4fed31a1c458-plugins-config-volume\") pod \"b62d2e16-32d3-46b2-8795-4fed31a1c458\" (UID: \"b62d2e16-32d3-46b2-8795-4fed31a1c458\") " Nov 04 00:32:36.901006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.900922 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62d2e16-32d3-46b2-8795-4fed31a1c458-kube-api-access-qjqtb\" (UniqueName: \"kubernetes.io/projected/b62d2e16-32d3-46b2-8795-4fed31a1c458-kube-api-access-qjqtb\") pod \"b62d2e16-32d3-46b2-8795-4fed31a1c458\" (UID: \"b62d2e16-32d3-46b2-8795-4fed31a1c458\") " Nov 04 00:32:36.901228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.901198 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62d2e16-32d3-46b2-8795-4fed31a1c458-plugins-config-volume" pod "b62d2e16-32d3-46b2-8795-4fed31a1c458" (UID: "b62d2e16-32d3-46b2-8795-4fed31a1c458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:32:36.901555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.901536 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmfmg\" (UniqueName: \"kubernetes.io/projected/127f6328-65eb-4bea-898b-f01e05d6a6dc-kube-api-access-mmfmg\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:36.901593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.901560 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62d2e16-32d3-46b2-8795-4fed31a1c458-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:36.901593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.901574 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/127f6328-65eb-4bea-898b-f01e05d6a6dc-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:36.903626 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:36.903605 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62d2e16-32d3-46b2-8795-4fed31a1c458-kube-api-access-qjqtb" pod "b62d2e16-32d3-46b2-8795-4fed31a1c458" (UID: "b62d2e16-32d3-46b2-8795-4fed31a1c458"). InnerVolumeSpecName "kube-api-access-qjqtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:37.003423 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.003385 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjqtb\" (UniqueName: \"kubernetes.io/projected/b62d2e16-32d3-46b2-8795-4fed31a1c458-kube-api-access-qjqtb\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:37.437573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.437524 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:37.446613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.446579 6208 generic.go:356] "Generic (PLEG): container finished" podID="127f6328-65eb-4bea-898b-f01e05d6a6dc" containerID="8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6" exitCode=0 Nov 04 00:32:37.446771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.446653 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" Nov 04 00:32:37.446771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.446661 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" event={"ID":"127f6328-65eb-4bea-898b-f01e05d6a6dc","Type":"ContainerDied","Data":"8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6"} Nov 04 00:32:37.446771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.446715 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" event={"ID":"127f6328-65eb-4bea-898b-f01e05d6a6dc","Type":"ContainerDied","Data":"239ee047da9b42ae5a6be3abbf403f23c9e8d74b1b82db637f4395674a129990"} Nov 04 00:32:37.446771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.446747 6208 scope.go:122] "RemoveContainer" containerID="8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6" Nov 04 00:32:37.448665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.448645 6208 generic.go:356] "Generic (PLEG): container finished" podID="b62d2e16-32d3-46b2-8795-4fed31a1c458" containerID="4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1" exitCode=0 Nov 04 00:32:37.448858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.448679 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" event={"ID":"b62d2e16-32d3-46b2-8795-4fed31a1c458","Type":"ContainerDied","Data":"4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1"} Nov 04 00:32:37.448981 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.448894 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" event={"ID":"b62d2e16-32d3-46b2-8795-4fed31a1c458","Type":"ContainerDied","Data":"2ce9a8e0f60e3350b0f3a6b03935a645a9d42f092475272dcb80b4fd213d41dd"} Nov 04 00:32:37.448981 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.448698 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" Nov 04 00:32:37.451904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.451882 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" event={"ID":"5b675c79-e14d-440f-b32b-24dd6bd05427","Type":"ContainerDied","Data":"5705a19a41f11f50fc6aae5aec410ed32a05a2ecfbc1e1a3318f269f9ba4f4bf"} Nov 04 00:32:37.452019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.451949 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66" Nov 04 00:32:37.461255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.461237 6208 scope.go:122] "RemoveContainer" containerID="8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6" Nov 04 00:32:37.461537 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:32:37.461518 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6\": container with ID starting with 8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6 not found: ID does not exist" containerID="8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6" Nov 04 00:32:37.461608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.461544 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6"} err="failed to get container status \"8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6\": rpc error: code = NotFound desc = could not find container \"8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6\": container with ID starting with 8592e06fac3daa2021582cc079c7f182b15ac732a352415aa68c5394d64a39a6 not found: ID does not exist" Nov 04 00:32:37.461608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.461567 6208 scope.go:122] "RemoveContainer" containerID="4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1" Nov 04 00:32:37.470803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.470776 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t"] Nov 04 00:32:37.473353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.473320 6208 scope.go:122] "RemoveContainer" containerID="4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1" Nov 04 00:32:37.473681 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:32:37.473655 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1\": container with ID starting with 4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1 not found: ID does not exist" containerID="4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1" Nov 04 00:32:37.473780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.473689 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1"} err="failed to get container status \"4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1\": rpc error: code = NotFound desc = could not find container \"4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1\": container with ID starting with 4e4410194ce57c878a760ca9b74ddb4943da0e9870b89cf4378f585c67d867b1 not found: ID does not exist" Nov 04 00:32:37.473780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.473709 6208 scope.go:122] "RemoveContainer" containerID="665111bb2f9b03e52d6529d54b6eb4401052ae8d0e20f5573c1726dfcec4fa64" Nov 04 00:32:37.476819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.476795 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t"] Nov 04 00:32:37.533496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.533141 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66"] Nov 04 00:32:37.539206 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.539171 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-2341ca-79d8f947cb-8mr66"] Nov 04 00:32:37.556575 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.556538 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9"] Nov 04 00:32:37.560082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.559997 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9"] Nov 04 00:32:37.750079 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.750023 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f6f292-68579547bc-g2dg9" podUID="b62d2e16-32d3-46b2-8795-4fed31a1c458" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.63:50052\" within 1s: context deadline exceeded" Nov 04 00:32:37.750278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:37.750024 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f6f292-559cff65d4-qvh4t" podUID="127f6328-65eb-4bea-898b-f01e05d6a6dc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.64:50052\" within 1s: context deadline exceeded" Nov 04 00:32:37.750851 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:32:37.750833 6208 logging.go:55] [core] [Channel #9546 SubChannel #9548]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.64:50052", ServerName: "10.133.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.64:50052: operation was canceled" Nov 04 00:32:37.750906 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:32:37.750834 6208 logging.go:55] [core] [Channel #9547 SubChannel #9549]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.63:50052", ServerName: "10.133.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.63:50052: operation was canceled" Nov 04 00:32:38.366612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:38.361725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:38.728571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:38.728532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:39.024393 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.024263 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:39.113468 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.113421 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" containerID="cri-o://7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3" gracePeriod=45 Nov 04 00:32:39.295987 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.295945 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127f6328-65eb-4bea-898b-f01e05d6a6dc" path="/var/lib/kubelet/pods/127f6328-65eb-4bea-898b-f01e05d6a6dc/volumes" Nov 04 00:32:39.296728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.296701 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b675c79-e14d-440f-b32b-24dd6bd05427" path="/var/lib/kubelet/pods/5b675c79-e14d-440f-b32b-24dd6bd05427/volumes" Nov 04 00:32:39.297290 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.297269 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62d2e16-32d3-46b2-8795-4fed31a1c458" path="/var/lib/kubelet/pods/b62d2e16-32d3-46b2-8795-4fed31a1c458/volumes" Nov 04 00:32:39.394547 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.394514 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 04 00:32:39.479321 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.479285 6208 generic.go:356] "Generic (PLEG): container finished" podID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerID="7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3" exitCode=0 Nov 04 00:32:39.479516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.479335 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" Nov 04 00:32:39.479516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.479368 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" event={"ID":"88cecb6a-ebee-4eb2-abac-1cc23957ae3d","Type":"ContainerDied","Data":"7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3"} Nov 04 00:32:39.479516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.479422 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" event={"ID":"88cecb6a-ebee-4eb2-abac-1cc23957ae3d","Type":"ContainerDied","Data":"0ae7114ea2445e1bb6c5f974646b47106ecd8c124335ae9ff5ba36c0181fe715"} Nov 04 00:32:39.479516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.479450 6208 scope.go:122] "RemoveContainer" containerID="7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3" Nov 04 00:32:39.493170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.493149 6208 scope.go:122] "RemoveContainer" containerID="7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3" Nov 04 00:32:39.493482 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:32:39.493461 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3\": container with ID starting with 7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3 not found: ID does not exist" containerID="7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3" Nov 04 00:32:39.493561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.493494 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3"} err="failed to get container status \"7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3\": rpc error: code = NotFound desc = could not find container \"7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3\": container with ID starting with 7d0792589ce1476149a2bfeb375a50e66a24c5d3120095eeecc86181ef1baee3 not found: ID does not exist" Nov 04 00:32:39.558738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.558651 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-kube-api-access-m4ltt\" (UniqueName: \"kubernetes.io/projected/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-kube-api-access-m4ltt\") pod \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " Nov 04 00:32:39.558905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.558776 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tmp-dir\") pod \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " Nov 04 00:32:39.558980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.558962 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-home\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-home\") pod \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " Nov 04 00:32:39.559115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.559001 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-dshm\") pod \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " Nov 04 00:32:39.559115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.559070 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-model-cache\") pod \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " Nov 04 00:32:39.559115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.559068 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tmp-dir" pod "88cecb6a-ebee-4eb2-abac-1cc23957ae3d" (UID: "88cecb6a-ebee-4eb2-abac-1cc23957ae3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:39.559306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.559130 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tls-certs\") pod \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\" (UID: \"88cecb6a-ebee-4eb2-abac-1cc23957ae3d\") " Nov 04 00:32:39.559306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.559239 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-home" pod "88cecb6a-ebee-4eb2-abac-1cc23957ae3d" (UID: "88cecb6a-ebee-4eb2-abac-1cc23957ae3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:39.559421 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.559397 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-model-cache" pod "88cecb6a-ebee-4eb2-abac-1cc23957ae3d" (UID: "88cecb6a-ebee-4eb2-abac-1cc23957ae3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:39.559821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.559794 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:39.559938 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.559826 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:39.559938 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.559841 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:39.561113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.561084 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-dshm" pod "88cecb6a-ebee-4eb2-abac-1cc23957ae3d" (UID: "88cecb6a-ebee-4eb2-abac-1cc23957ae3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:32:39.561201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.561144 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-kube-api-access-m4ltt" pod "88cecb6a-ebee-4eb2-abac-1cc23957ae3d" (UID: "88cecb6a-ebee-4eb2-abac-1cc23957ae3d"). InnerVolumeSpecName "kube-api-access-m4ltt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:39.561633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.561613 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tls-certs" pod "88cecb6a-ebee-4eb2-abac-1cc23957ae3d" (UID: "88cecb6a-ebee-4eb2-abac-1cc23957ae3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:32:39.661598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.661566 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:39.661598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.661594 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:39.661598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.661605 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4ltt\" (UniqueName: \"kubernetes.io/projected/88cecb6a-ebee-4eb2-abac-1cc23957ae3d-kube-api-access-m4ltt\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:39.806722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.806692 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp"] Nov 04 00:32:39.811195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.811128 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp"] Nov 04 00:32:39.856640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:39.856614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:40.335767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:40.335722 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f6f292/simulator-w0-f6f292-kserve-74858bcf8-rjkcp" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 00:32:40.631075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:40.630968 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:41.296849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:41.296812 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" path="/var/lib/kubelet/pods/88cecb6a-ebee-4eb2-abac-1cc23957ae3d/volumes" Nov 04 00:32:41.433737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:41.433701 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:42.109700 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:42.109662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:42.235062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:42.234462 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:42.641735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:42.641705 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:43.004015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:43.003933 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:43.237040 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:43.237005 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:43.699863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:43.699830 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:44.038274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:44.038236 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:44.865684 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:44.865644 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:45.619179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:45.619124 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:46.465535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:46.465496 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:47.089944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:47.089897 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:47.453383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:47.453294 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:48.394059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:48.394003 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:49.221594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:49.221552 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:49.947361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:49.947325 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:50.822758 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:50.822708 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:51.724729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:51.724694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:52.460363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:52.460331 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:53.191218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:53.191172 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:53.718993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:53.718955 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:53.983723 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:53.983630 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:54.785861 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:54.785798 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:55.362831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.362787 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv"] Nov 04 00:32:55.363117 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.363084 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" podUID="204d311a-3628-48a3-98f1-36cf66bc12bc" containerName="maas-api" containerID="cri-o://5939f7df0e40d1770a11e9c8e0b5dde2a19e7cef0434a2f802a894597304bc07" gracePeriod=30 Nov 04 00:32:55.405693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.405660 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:55.763338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.763282 6208 generic.go:356] "Generic (PLEG): container finished" podID="204d311a-3628-48a3-98f1-36cf66bc12bc" containerID="5939f7df0e40d1770a11e9c8e0b5dde2a19e7cef0434a2f802a894597304bc07" exitCode=0 Nov 04 00:32:55.763338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.763324 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" event={"ID":"204d311a-3628-48a3-98f1-36cf66bc12bc","Type":"ContainerDied","Data":"5939f7df0e40d1770a11e9c8e0b5dde2a19e7cef0434a2f802a894597304bc07"} Nov 04 00:32:55.799858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.799822 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:55.799858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.799850 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 04 00:32:55.875701 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.875664 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204d311a-3628-48a3-98f1-36cf66bc12bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204d311a-3628-48a3-98f1-36cf66bc12bc-openshift-service-ca\") pod \"204d311a-3628-48a3-98f1-36cf66bc12bc\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " Nov 04 00:32:55.875879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.875811 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-metrics-certs\") pod \"204d311a-3628-48a3-98f1-36cf66bc12bc\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " Nov 04 00:32:55.875949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.875925 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-maas-api-tls\") pod \"204d311a-3628-48a3-98f1-36cf66bc12bc\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " Nov 04 00:32:55.875988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.875957 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204d311a-3628-48a3-98f1-36cf66bc12bc-kube-api-access-2ndq5\" (UniqueName: \"kubernetes.io/projected/204d311a-3628-48a3-98f1-36cf66bc12bc-kube-api-access-2ndq5\") pod \"204d311a-3628-48a3-98f1-36cf66bc12bc\" (UID: \"204d311a-3628-48a3-98f1-36cf66bc12bc\") " Nov 04 00:32:55.876136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.876107 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204d311a-3628-48a3-98f1-36cf66bc12bc-openshift-service-ca" pod "204d311a-3628-48a3-98f1-36cf66bc12bc" (UID: "204d311a-3628-48a3-98f1-36cf66bc12bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:32:55.876430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.876412 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/204d311a-3628-48a3-98f1-36cf66bc12bc-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:55.878008 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.877980 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-maas-api-tls" pod "204d311a-3628-48a3-98f1-36cf66bc12bc" (UID: "204d311a-3628-48a3-98f1-36cf66bc12bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:32:55.878159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.878051 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204d311a-3628-48a3-98f1-36cf66bc12bc-kube-api-access-2ndq5" pod "204d311a-3628-48a3-98f1-36cf66bc12bc" (UID: "204d311a-3628-48a3-98f1-36cf66bc12bc"). InnerVolumeSpecName "kube-api-access-2ndq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:32:55.878205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.878164 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-metrics-certs" pod "204d311a-3628-48a3-98f1-36cf66bc12bc" (UID: "204d311a-3628-48a3-98f1-36cf66bc12bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:32:55.977897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.977851 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:55.977897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.977898 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/204d311a-3628-48a3-98f1-36cf66bc12bc-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:55.978256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:55.977913 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndq5\" (UniqueName: \"kubernetes.io/projected/204d311a-3628-48a3-98f1-36cf66bc12bc-kube-api-access-2ndq5\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:32:56.517445 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:56.517405 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:56.771919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:56.771839 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" Nov 04 00:32:56.771919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:56.771876 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv" event={"ID":"204d311a-3628-48a3-98f1-36cf66bc12bc","Type":"ContainerDied","Data":"358fd090045037914cdd2fd29ce3e834ac49802f4701163878c5fa6d47cedbd2"} Nov 04 00:32:56.772173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:56.771935 6208 scope.go:122] "RemoveContainer" containerID="5939f7df0e40d1770a11e9c8e0b5dde2a19e7cef0434a2f802a894597304bc07" Nov 04 00:32:56.797003 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:56.796975 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv"] Nov 04 00:32:56.800858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:56.800833 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f6f292-66b88f46c9-8fsnv"] Nov 04 00:32:57.294596 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:57.294564 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204d311a-3628-48a3-98f1-36cf66bc12bc" path="/var/lib/kubelet/pods/204d311a-3628-48a3-98f1-36cf66bc12bc/volumes" Nov 04 00:32:57.322226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:57.322194 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:57.736099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:57.736063 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:58.225045 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:58.224990 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:59.023582 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:59.023546 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:59.815439 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:59.815406 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:32:59.914644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:32:59.914613 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:00.500596 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:00.500558 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:00.946613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:00.946514 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:01.908490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:01.908405 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:02.005575 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:02.005536 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:02.656634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:02.656600 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:03.380221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:03.380182 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:03.991734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:03.991697 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:04.185043 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:04.184994 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:04.285152 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:04.285058 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:04.740721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:04.740680 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:05.062390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:05.062296 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:05.448982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:05.448879 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:05.706635 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:33:05.706600 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:33:06.414184 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:06.414148 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:07.243142 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:07.243101 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:07.847278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:07.847228 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:07.953342 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:07.953306 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:08.769972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:08.769913 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:09.622556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:09.622518 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:10.416247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:10.416207 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:10.747116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:10.747080 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:10.850950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:10.850908 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:11.221689 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:11.221647 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:12.091422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:12.091377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:13.054380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:13.054343 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:13.770571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:13.770529 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:14.125623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:14.125532 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:14.798378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:14.798342 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:14.899166 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:14.899134 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:14.999737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:14.999697 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:15.287505 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:15.287469 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:15.601853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:15.601769 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8"] Nov 04 00:33:15.602097 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:15.602070 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" containerID="cri-o://b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472" gracePeriod=30 Nov 04 00:33:15.642601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:15.642567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:15.939268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:15.939185 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:16.112475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:16.112429 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:16.849651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:16.849611 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:17.680512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:17.680473 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:19.439847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:19.439767 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:20.134501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:20.134438 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:20.895231 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:20.895201 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 04 00:33:20.975903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:20.975863 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:20.983113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:20.983076 6208 generic.go:356] "Generic (PLEG): container finished" podID="84c01b66-326c-41c4-b612-5551e8af6e85" containerID="b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472" exitCode=0 Nov 04 00:33:20.983302 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:20.983179 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" Nov 04 00:33:20.983302 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:20.983175 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" event={"ID":"84c01b66-326c-41c4-b612-5551e8af6e85","Type":"ContainerDied","Data":"b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472"} Nov 04 00:33:20.983419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:20.983322 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8" event={"ID":"84c01b66-326c-41c4-b612-5551e8af6e85","Type":"ContainerDied","Data":"79580ca9657aa7436366f27baff17e2888e1980b5461ee97af69db765a80f978"} Nov 04 00:33:20.983419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:20.983347 6208 scope.go:122] "RemoveContainer" containerID="b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472" Nov 04 00:33:20.996718 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:20.996694 6208 scope.go:122] "RemoveContainer" containerID="62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293" Nov 04 00:33:21.011428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.011401 6208 scope.go:122] "RemoveContainer" containerID="b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472" Nov 04 00:33:21.011768 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:33:21.011746 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472\": container with ID starting with b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472 not found: ID does not exist" containerID="b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472" Nov 04 00:33:21.011842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.011777 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472"} err="failed to get container status \"b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472\": rpc error: code = NotFound desc = could not find container \"b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472\": container with ID starting with b34f1fceb25b8374d559906a4a03c6b33c8e77478dc36ea0afbe85285aed0472 not found: ID does not exist" Nov 04 00:33:21.011842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.011808 6208 scope.go:122] "RemoveContainer" containerID="62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293" Nov 04 00:33:21.012196 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:33:21.012162 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293\": container with ID starting with 62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293 not found: ID does not exist" containerID="62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293" Nov 04 00:33:21.012299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.012205 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293"} err="failed to get container status \"62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293\": rpc error: code = NotFound desc = could not find container \"62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293\": container with ID starting with 62460f2771c3485ad70c1fb0756c97806928b8fb4f38f9d34a1effe35c5e9293 not found: ID does not exist" Nov 04 00:33:21.076406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076363 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-certs\") pod \"84c01b66-326c-41c4-b612-5551e8af6e85\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " Nov 04 00:33:21.076406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076403 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-envoy\") pod \"84c01b66-326c-41c4-b612-5551e8af6e85\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " Nov 04 00:33:21.076675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076449 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-socket\") pod \"84c01b66-326c-41c4-b612-5551e8af6e85\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " Nov 04 00:33:21.076675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076480 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-data\") pod \"84c01b66-326c-41c4-b612-5551e8af6e85\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " Nov 04 00:33:21.076675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076514 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-kube-api-access-b6wsw\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-kube-api-access-b6wsw\") pod \"84c01b66-326c-41c4-b612-5551e8af6e85\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " Nov 04 00:33:21.076675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076585 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-istio-token\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-istio-token\") pod \"84c01b66-326c-41c4-b612-5551e8af6e85\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " Nov 04 00:33:21.076913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076682 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84c01b66-326c-41c4-b612-5551e8af6e85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c01b66-326c-41c4-b612-5551e8af6e85-istio-podinfo\") pod \"84c01b66-326c-41c4-b612-5551e8af6e85\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " Nov 04 00:33:21.076913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076716 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c01b66-326c-41c4-b612-5551e8af6e85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c01b66-326c-41c4-b612-5551e8af6e85-istiod-ca-cert\") pod \"84c01b66-326c-41c4-b612-5551e8af6e85\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " Nov 04 00:33:21.076913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076737 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-certs" pod "84c01b66-326c-41c4-b612-5551e8af6e85" (UID: "84c01b66-326c-41c4-b612-5551e8af6e85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:33:21.076913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076854 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-credential-socket\") pod \"84c01b66-326c-41c4-b612-5551e8af6e85\" (UID: \"84c01b66-326c-41c4-b612-5551e8af6e85\") " Nov 04 00:33:21.077227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.076908 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-socket" pod "84c01b66-326c-41c4-b612-5551e8af6e85" (UID: "84c01b66-326c-41c4-b612-5551e8af6e85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:33:21.077468 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.077414 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:21.077468 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.077440 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:21.077779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.077751 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-credential-socket" pod "84c01b66-326c-41c4-b612-5551e8af6e85" (UID: "84c01b66-326c-41c4-b612-5551e8af6e85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:33:21.078292 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.077814 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-data" pod "84c01b66-326c-41c4-b612-5551e8af6e85" (UID: "84c01b66-326c-41c4-b612-5551e8af6e85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:33:21.098698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.095158 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c01b66-326c-41c4-b612-5551e8af6e85-istiod-ca-cert" pod "84c01b66-326c-41c4-b612-5551e8af6e85" (UID: "84c01b66-326c-41c4-b612-5551e8af6e85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:33:21.105898 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.105099 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb"] Nov 04 00:33:21.110594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.107359 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-kube-api-access-b6wsw" pod "84c01b66-326c-41c4-b612-5551e8af6e85" (UID: "84c01b66-326c-41c4-b612-5551e8af6e85"). InnerVolumeSpecName "kube-api-access-b6wsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:33:21.110594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.107482 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-istio-token" pod "84c01b66-326c-41c4-b612-5551e8af6e85" (UID: "84c01b66-326c-41c4-b612-5551e8af6e85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:33:21.110594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.107791 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-envoy" pod "84c01b66-326c-41c4-b612-5551e8af6e85" (UID: "84c01b66-326c-41c4-b612-5551e8af6e85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:33:21.111744 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.111692 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84c01b66-326c-41c4-b612-5551e8af6e85-istio-podinfo" pod "84c01b66-326c-41c4-b612-5551e8af6e85" (UID: "84c01b66-326c-41c4-b612-5551e8af6e85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:33:21.179021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.178937 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c01b66-326c-41c4-b612-5551e8af6e85-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:21.179021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.178975 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c01b66-326c-41c4-b612-5551e8af6e85-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:21.179021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.178989 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:21.179021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.179004 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:21.179021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.179019 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c01b66-326c-41c4-b612-5551e8af6e85-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:21.179021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.179051 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6wsw\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-kube-api-access-b6wsw\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:21.179021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.179067 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c01b66-326c-41c4-b612-5551e8af6e85-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:21.186191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.186157 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:21.324069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.324017 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8"] Nov 04 00:33:21.328741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.328709 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2b1f66-openshift-default-7dfd96486f-9l6k8"] Nov 04 00:33:21.695312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.695275 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:21.927588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:21.927498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:22.836858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:22.836800 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:23.294859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:23.294820 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" path="/var/lib/kubelet/pods/84c01b66-326c-41c4-b612-5551e8af6e85/volumes" Nov 04 00:33:23.556119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:23.556016 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:24.373985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:24.373950 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:25.223987 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:25.223943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:26.051667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:26.051630 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:26.887462 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:26.887422 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:27.364717 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:27.364669 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:27.467536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:27.467498 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:27.935173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:27.935140 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:28.739353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:28.739312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:29.529409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:29.529370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:30.321411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:30.321378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:30.490610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:30.490568 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:31.048748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:31.048710 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:32.028822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:32.028775 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:32.926584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:32.926551 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:33.827350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:33.827310 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:34.729270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:34.729231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:35.450204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:35.450154 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:35.705162 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:33:35.705078 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:33:36.155172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.155106 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" containerID="cri-o://31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d" gracePeriod=45 Nov 04 00:33:36.171378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.171348 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:36.420688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.420630 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 04 00:33:36.544506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.544472 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tmp-dir\") pod \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " Nov 04 00:33:36.544937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.544559 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-model-cache\") pod \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " Nov 04 00:33:36.544937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.544606 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-dshm\") pod \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " Nov 04 00:33:36.544937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.544643 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-kube-api-access-gwn69\" (UniqueName: \"kubernetes.io/projected/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-kube-api-access-gwn69\") pod \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " Nov 04 00:33:36.544937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.544695 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tls-certs\") pod \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " Nov 04 00:33:36.544937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.544802 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-home\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-home\") pod \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\" (UID: \"46d2d360-21ba-4c73-b6a1-fe28276a6f2f\") " Nov 04 00:33:36.573279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.573207 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tmp-dir" pod "46d2d360-21ba-4c73-b6a1-fe28276a6f2f" (UID: "46d2d360-21ba-4c73-b6a1-fe28276a6f2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:33:36.573442 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.573243 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-model-cache" pod "46d2d360-21ba-4c73-b6a1-fe28276a6f2f" (UID: "46d2d360-21ba-4c73-b6a1-fe28276a6f2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:33:36.573442 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.573276 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-home" pod "46d2d360-21ba-4c73-b6a1-fe28276a6f2f" (UID: "46d2d360-21ba-4c73-b6a1-fe28276a6f2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:33:36.575169 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.575144 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tls-certs" pod "46d2d360-21ba-4c73-b6a1-fe28276a6f2f" (UID: "46d2d360-21ba-4c73-b6a1-fe28276a6f2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:33:36.575289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.575178 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-dshm" pod "46d2d360-21ba-4c73-b6a1-fe28276a6f2f" (UID: "46d2d360-21ba-4c73-b6a1-fe28276a6f2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:33:36.575289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.575222 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-kube-api-access-gwn69" pod "46d2d360-21ba-4c73-b6a1-fe28276a6f2f" (UID: "46d2d360-21ba-4c73-b6a1-fe28276a6f2f"). InnerVolumeSpecName "kube-api-access-gwn69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:33:36.646095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.646054 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:36.646095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.646089 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:36.646095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.646102 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:36.646358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.646114 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:36.646358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.646138 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwn69\" (UniqueName: \"kubernetes.io/projected/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-kube-api-access-gwn69\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:36.646358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.646149 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d2d360-21ba-4c73-b6a1-fe28276a6f2f-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:36.980148 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:36.980109 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:37.141465 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.141428 6208 generic.go:356] "Generic (PLEG): container finished" podID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerID="31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d" exitCode=0 Nov 04 00:33:37.141639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.141471 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" Nov 04 00:33:37.141639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.141513 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" event={"ID":"46d2d360-21ba-4c73-b6a1-fe28276a6f2f","Type":"ContainerDied","Data":"31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d"} Nov 04 00:33:37.141639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.141558 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb" event={"ID":"46d2d360-21ba-4c73-b6a1-fe28276a6f2f","Type":"ContainerDied","Data":"06d01716ff729596fa3c21bef916b4bb57997c7e17d465f1c4adf15fbd039bf8"} Nov 04 00:33:37.141639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.141579 6208 scope.go:122] "RemoveContainer" containerID="31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d" Nov 04 00:33:37.153718 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.153701 6208 scope.go:122] "RemoveContainer" containerID="31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d" Nov 04 00:33:37.153990 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:33:37.153972 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d\": container with ID starting with 31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d not found: ID does not exist" containerID="31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d" Nov 04 00:33:37.154053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.154001 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d"} err="failed to get container status \"31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d\": rpc error: code = NotFound desc = could not find container \"31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d\": container with ID starting with 31a16d02dde8cb0e83c852145f5cec2cfaf64096552679c31a6b759baa33a03d not found: ID does not exist" Nov 04 00:33:37.166286 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.166262 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb"] Nov 04 00:33:37.172789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.172754 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5cb349/distinct-2-w1-5cb349-kserve-7bdd58cf9f-v2mrb"] Nov 04 00:33:37.294605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.294521 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" path="/var/lib/kubelet/pods/46d2d360-21ba-4c73-b6a1-fe28276a6f2f/volumes" Nov 04 00:33:37.436937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.436906 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:37.661106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:37.660998 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:38.437047 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:38.436991 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:39.283513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:39.283470 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:39.991942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:39.991904 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:40.823591 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:40.823556 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:41.683317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:41.683279 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:42.509788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:42.505937 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:43.194298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:43.194254 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:43.892142 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:43.892106 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:44.635101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:44.635059 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:44.734868 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:44.734832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:45.351658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:45.351614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:46.128688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:46.128655 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:46.998959 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:46.998916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:47.755839 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:47.755797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:48.763826 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:48.763788 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:49.691970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:49.691887 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:50.663525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:50.663489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:51.465667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:51.465591 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:52.257223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:52.257184 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:53.134813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:53.134778 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:54.014430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:54.014388 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:54.722495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:54.722459 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:55.459199 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:55.459160 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:56.030999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.030959 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf"] Nov 04 00:33:56.031262 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.031237 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" podUID="1824481f-5659-4932-ab0b-6726f418b3a0" containerName="maas-api" containerID="cri-o://dc23a685b94afb0240a3653054a8aace10116a26ab254ec678d35f0465098cf3" gracePeriod=30 Nov 04 00:33:56.069212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.069179 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:56.399973 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.399939 6208 generic.go:356] "Generic (PLEG): container finished" podID="1824481f-5659-4932-ab0b-6726f418b3a0" containerID="dc23a685b94afb0240a3653054a8aace10116a26ab254ec678d35f0465098cf3" exitCode=0 Nov 04 00:33:56.400149 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.400008 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" event={"ID":"1824481f-5659-4932-ab0b-6726f418b3a0","Type":"ContainerDied","Data":"dc23a685b94afb0240a3653054a8aace10116a26ab254ec678d35f0465098cf3"} Nov 04 00:33:56.426179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.426149 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 04 00:33:56.461349 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.461312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:56.515407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.515366 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1824481f-5659-4932-ab0b-6726f418b3a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1824481f-5659-4932-ab0b-6726f418b3a0-openshift-service-ca\") pod \"1824481f-5659-4932-ab0b-6726f418b3a0\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " Nov 04 00:33:56.515610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.515427 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1824481f-5659-4932-ab0b-6726f418b3a0-kube-api-access-pt8bm\" (UniqueName: \"kubernetes.io/projected/1824481f-5659-4932-ab0b-6726f418b3a0-kube-api-access-pt8bm\") pod \"1824481f-5659-4932-ab0b-6726f418b3a0\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " Nov 04 00:33:56.515610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.515501 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-maas-api-tls\") pod \"1824481f-5659-4932-ab0b-6726f418b3a0\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " Nov 04 00:33:56.515736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.515668 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-metrics-certs\") pod \"1824481f-5659-4932-ab0b-6726f418b3a0\" (UID: \"1824481f-5659-4932-ab0b-6726f418b3a0\") " Nov 04 00:33:56.515816 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.515785 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1824481f-5659-4932-ab0b-6726f418b3a0-openshift-service-ca" pod "1824481f-5659-4932-ab0b-6726f418b3a0" (UID: "1824481f-5659-4932-ab0b-6726f418b3a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:33:56.516195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.516174 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1824481f-5659-4932-ab0b-6726f418b3a0-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:56.517758 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.517735 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-maas-api-tls" pod "1824481f-5659-4932-ab0b-6726f418b3a0" (UID: "1824481f-5659-4932-ab0b-6726f418b3a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:33:56.517830 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.517763 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-metrics-certs" pod "1824481f-5659-4932-ab0b-6726f418b3a0" (UID: "1824481f-5659-4932-ab0b-6726f418b3a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:33:56.517830 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.517772 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1824481f-5659-4932-ab0b-6726f418b3a0-kube-api-access-pt8bm" pod "1824481f-5659-4932-ab0b-6726f418b3a0" (UID: "1824481f-5659-4932-ab0b-6726f418b3a0"). InnerVolumeSpecName "kube-api-access-pt8bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:33:56.616999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.616960 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:56.616999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.616994 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt8bm\" (UniqueName: \"kubernetes.io/projected/1824481f-5659-4932-ab0b-6726f418b3a0-kube-api-access-pt8bm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:56.616999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:56.617005 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1824481f-5659-4932-ab0b-6726f418b3a0-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:33:57.226506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:57.226471 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:57.408149 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:57.408106 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" Nov 04 00:33:57.408360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:57.408103 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf" event={"ID":"1824481f-5659-4932-ab0b-6726f418b3a0","Type":"ContainerDied","Data":"cba9298945087976de74a8332f651cceacf4cbba3c6a4e7262105125f263929a"} Nov 04 00:33:57.408360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:57.408269 6208 scope.go:122] "RemoveContainer" containerID="dc23a685b94afb0240a3653054a8aace10116a26ab254ec678d35f0465098cf3" Nov 04 00:33:57.429095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:57.429058 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf"] Nov 04 00:33:57.433072 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:57.433047 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5cb349-7589f869c-jxvsf"] Nov 04 00:33:57.885145 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:57.885105 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:58.122368 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:58.122327 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:58.308472 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:58.308441 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:58.593832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:58.593747 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:58.742028 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:58.741993 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:58.961636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:58.961603 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:59.081761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:59.081724 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:33:59.298905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:59.298822 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1824481f-5659-4932-ab0b-6726f418b3a0" path="/var/lib/kubelet/pods/1824481f-5659-4932-ab0b-6726f418b3a0/volumes" Nov 04 00:33:59.946542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:33:59.946508 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:00.727961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:00.727924 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:00.930463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:00.930427 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:01.545227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:01.545188 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:02.344350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:02.344013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:03.207485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:03.207453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:03.311476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:03.311436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:03.724342 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:03.724303 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:04.109617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:04.109529 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:04.237501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:04.237440 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:04.364685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:04.364593 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:05.206056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:05.204853 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:05.707314 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:34:05.707276 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:34:06.053598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:06.053511 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:06.823643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:06.823604 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:07.212123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:07.212084 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:07.312260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:07.312216 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:07.621236 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:07.621137 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:08.350237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:08.350198 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:09.142905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:09.142873 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:09.911451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:09.911412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:10.813052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:10.813007 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:11.425759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:11.425717 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:12.129502 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:12.129457 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:12.832540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:12.832503 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:13.620193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:13.620148 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:14.342809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:14.342771 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:15.259776 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:15.259737 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:16.039633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:16.039592 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:16.842846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:16.842810 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:17.642068 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:17.642006 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:18.439383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:18.439350 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:19.327369 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:19.327337 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:20.227183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:20.227142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:20.936668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:20.936625 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:21.637343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:21.637302 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:22.366667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:22.366630 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:23.150745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:23.150711 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:23.941553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:23.941513 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:24.735930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:24.735899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:25.452863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:25.452824 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:26.308293 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:26.308250 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:26.842541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:26.842506 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:27.237606 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:27.237560 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:28.118412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:28.118370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:28.928824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:28.928769 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:29.751092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:29.751057 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:30.953848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:30.953802 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:31.785921 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:31.785886 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:32.075801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:32.075694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:32.552102 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:32.552065 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:32.769731 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:32.769692 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:33.125174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:33.125136 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:33.224870 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:33.224834 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:33.653857 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:33.653812 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:34.431046 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:34.430997 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:35.249809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:35.249776 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:35.708228 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:34:35.708193 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:34:36.068091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:36.067981 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:36.851163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:36.851123 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:37.801757 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:37.801722 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:38.634666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:38.634625 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:38.925293 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:38.925210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:39.024507 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:39.024464 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:39.551320 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:39.551289 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:40.260758 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:40.260719 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:40.597883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:40.597796 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:41.049958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:41.049918 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:41.941158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:41.941125 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:42.729120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:42.729085 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:43.024606 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:43.024524 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:34:43.028495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:43.028470 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:34:43.443757 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:43.443665 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:44.330848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:44.330807 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:45.152494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:45.152443 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:46.132984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:46.132935 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:46.728204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:46.728167 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:47.037517 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:47.037421 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:47.532851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:47.532815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:48.477524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:48.477481 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:48.776223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:48.776131 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:49.307305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:49.307275 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:50.027180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:50.027143 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:50.757212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:50.757181 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:51.312930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.312879 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq"] Nov 04 00:34:51.312930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.313132 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" podUID="5a7f005b-f784-4e32-9eee-6582056d16fe" containerName="maas-api" containerID="cri-o://4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87" gracePeriod=30 Nov 04 00:34:51.361160 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.361119 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:51.675802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.675777 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:34:51.716373 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.716336 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:51.761304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.761271 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-maas-api-tls\") pod \"5a7f005b-f784-4e32-9eee-6582056d16fe\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " Nov 04 00:34:51.761477 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.761357 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-metrics-certs\") pod \"5a7f005b-f784-4e32-9eee-6582056d16fe\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " Nov 04 00:34:51.761477 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.761395 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7f005b-f784-4e32-9eee-6582056d16fe-kube-api-access-286pl\" (UniqueName: \"kubernetes.io/projected/5a7f005b-f784-4e32-9eee-6582056d16fe-kube-api-access-286pl\") pod \"5a7f005b-f784-4e32-9eee-6582056d16fe\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " Nov 04 00:34:51.761477 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.761450 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7f005b-f784-4e32-9eee-6582056d16fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7f005b-f784-4e32-9eee-6582056d16fe-openshift-service-ca\") pod \"5a7f005b-f784-4e32-9eee-6582056d16fe\" (UID: \"5a7f005b-f784-4e32-9eee-6582056d16fe\") " Nov 04 00:34:51.761924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.761889 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7f005b-f784-4e32-9eee-6582056d16fe-openshift-service-ca" pod "5a7f005b-f784-4e32-9eee-6582056d16fe" (UID: "5a7f005b-f784-4e32-9eee-6582056d16fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:34:51.762208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.762185 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7f005b-f784-4e32-9eee-6582056d16fe-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:34:51.763528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.763497 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-metrics-certs" pod "5a7f005b-f784-4e32-9eee-6582056d16fe" (UID: "5a7f005b-f784-4e32-9eee-6582056d16fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:34:51.763633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.763536 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-maas-api-tls" pod "5a7f005b-f784-4e32-9eee-6582056d16fe" (UID: "5a7f005b-f784-4e32-9eee-6582056d16fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:34:51.763633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.763585 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7f005b-f784-4e32-9eee-6582056d16fe-kube-api-access-286pl" pod "5a7f005b-f784-4e32-9eee-6582056d16fe" (UID: "5a7f005b-f784-4e32-9eee-6582056d16fe"). InnerVolumeSpecName "kube-api-access-286pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:34:51.859913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.859875 6208 generic.go:356] "Generic (PLEG): container finished" podID="5a7f005b-f784-4e32-9eee-6582056d16fe" containerID="4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87" exitCode=0 Nov 04 00:34:51.860126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.859923 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" Nov 04 00:34:51.860126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.859958 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" event={"ID":"5a7f005b-f784-4e32-9eee-6582056d16fe","Type":"ContainerDied","Data":"4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87"} Nov 04 00:34:51.860126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.860021 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq" event={"ID":"5a7f005b-f784-4e32-9eee-6582056d16fe","Type":"ContainerDied","Data":"745c8afcf16706eca3fc4fbdf2f7c2edb8bb4dd3c4a435fd8198bdd521880eb5"} Nov 04 00:34:51.860126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.860103 6208 scope.go:122] "RemoveContainer" containerID="4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87" Nov 04 00:34:51.863358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.863332 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:34:51.863358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.863354 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a7f005b-f784-4e32-9eee-6582056d16fe-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:34:51.863558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.863365 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-286pl\" (UniqueName: \"kubernetes.io/projected/5a7f005b-f784-4e32-9eee-6582056d16fe-kube-api-access-286pl\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:34:51.871805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.871785 6208 scope.go:122] "RemoveContainer" containerID="4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87" Nov 04 00:34:51.872068 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:34:51.872049 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87\": container with ID starting with 4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87 not found: ID does not exist" containerID="4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87" Nov 04 00:34:51.872119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.872077 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87"} err="failed to get container status \"4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87\": rpc error: code = NotFound desc = could not find container \"4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87\": container with ID starting with 4c20b4d8599cc9e90ec6de266bf0ce73281971dc7812b710f221f1b6b8312c87 not found: ID does not exist" Nov 04 00:34:51.884813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.884791 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq"] Nov 04 00:34:51.887522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:51.887501 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-86deca-5746598546-gxrcq"] Nov 04 00:34:52.627952 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:52.627916 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:53.295088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:53.295056 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7f005b-f784-4e32-9eee-6582056d16fe" path="/var/lib/kubelet/pods/5a7f005b-f784-4e32-9eee-6582056d16fe/volumes" Nov 04 00:34:53.337988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:53.337942 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:54.088932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:54.088894 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:54.211060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:54.202952 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:54.347104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:54.347008 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:55.031221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:55.031185 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:55.833492 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:55.833457 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:56.734912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:56.734876 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:57.445802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:57.445765 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:57.548895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:57.548851 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:57.861511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:57.861475 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:58.433143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:58.433104 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:34:59.342498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:34:59.342457 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:00.234162 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:00.234122 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:01.007763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:01.007723 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:01.835555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:01.835518 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:02.152256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:02.152154 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:02.697163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:02.697131 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:02.844982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:02.844935 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:03.695080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:03.695044 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:04.423884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:04.423851 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:04.747510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:04.747469 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:04.897457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:04.897423 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:05.198078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:05.197960 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:05.663468 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:05.663415 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:05.705680 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:35:05.705648 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:35:05.950950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:05.950861 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:06.633330 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:06.633289 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:07.422093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:07.422052 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:08.077449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:08.077411 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:08.267873 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:08.267840 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:08.427103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:08.424882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:09.415668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:09.415630 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:10.136475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:10.136432 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:10.324963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:10.324919 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:10.864709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:10.864674 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:11.171724 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:11.171642 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:12.001418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:12.001378 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79"] Nov 04 00:35:12.076866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:12.076832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:12.195887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:12.195847 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:13.096534 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:13.095960 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:13.661133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:13.659259 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq"] Nov 04 00:35:13.661133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:13.659500 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" podUID="da7d10bd-27ea-429e-8730-1b67bb6e7e46" containerName="maas-api" containerID="cri-o://4c0ace1f1efc9c0e7c762eeb379142967c6489cf117281650e998f8d58439623" gracePeriod=30 Nov 04 00:35:13.710691 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:13.710655 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:13.998007 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:13.974162 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:14.035237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.035208 6208 generic.go:356] "Generic (PLEG): container finished" podID="da7d10bd-27ea-429e-8730-1b67bb6e7e46" containerID="4c0ace1f1efc9c0e7c762eeb379142967c6489cf117281650e998f8d58439623" exitCode=0 Nov 04 00:35:14.035380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.035279 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" event={"ID":"da7d10bd-27ea-429e-8730-1b67bb6e7e46","Type":"ContainerDied","Data":"4c0ace1f1efc9c0e7c762eeb379142967c6489cf117281650e998f8d58439623"} Nov 04 00:35:14.035380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.035330 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" event={"ID":"da7d10bd-27ea-429e-8730-1b67bb6e7e46","Type":"ContainerDied","Data":"7a023ad64c1609af3e6b6f85a74cca8cf24b3dda0e29f96d3b608a46201289af"} Nov 04 00:35:14.035380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.035348 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a023ad64c1609af3e6b6f85a74cca8cf24b3dda0e29f96d3b608a46201289af" Nov 04 00:35:14.039797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.039776 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:35:14.130318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.130283 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7d10bd-27ea-429e-8730-1b67bb6e7e46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d10bd-27ea-429e-8730-1b67bb6e7e46-openshift-service-ca\") pod \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " Nov 04 00:35:14.130523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.130328 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7d10bd-27ea-429e-8730-1b67bb6e7e46-kube-api-access-zr2r5\" (UniqueName: \"kubernetes.io/projected/da7d10bd-27ea-429e-8730-1b67bb6e7e46-kube-api-access-zr2r5\") pod \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " Nov 04 00:35:14.130523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.130359 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-maas-api-tls\") pod \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " Nov 04 00:35:14.130523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.130425 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-metrics-certs\") pod \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\" (UID: \"da7d10bd-27ea-429e-8730-1b67bb6e7e46\") " Nov 04 00:35:14.130715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.130648 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7d10bd-27ea-429e-8730-1b67bb6e7e46-openshift-service-ca" pod "da7d10bd-27ea-429e-8730-1b67bb6e7e46" (UID: "da7d10bd-27ea-429e-8730-1b67bb6e7e46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:35:14.131052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.131008 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7d10bd-27ea-429e-8730-1b67bb6e7e46-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:14.131552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.131522 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:14.132830 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.132801 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-maas-api-tls" pod "da7d10bd-27ea-429e-8730-1b67bb6e7e46" (UID: "da7d10bd-27ea-429e-8730-1b67bb6e7e46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:35:14.132992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.132975 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7d10bd-27ea-429e-8730-1b67bb6e7e46-kube-api-access-zr2r5" pod "da7d10bd-27ea-429e-8730-1b67bb6e7e46" (UID: "da7d10bd-27ea-429e-8730-1b67bb6e7e46"). InnerVolumeSpecName "kube-api-access-zr2r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:35:14.133100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.132983 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-metrics-certs" pod "da7d10bd-27ea-429e-8730-1b67bb6e7e46" (UID: "da7d10bd-27ea-429e-8730-1b67bb6e7e46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:35:14.231804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.231767 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2r5\" (UniqueName: \"kubernetes.io/projected/da7d10bd-27ea-429e-8730-1b67bb6e7e46-kube-api-access-zr2r5\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:14.231804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.231798 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:14.231804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.231811 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7d10bd-27ea-429e-8730-1b67bb6e7e46-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:14.960444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:14.960411 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:15.047715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:15.047683 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq" Nov 04 00:35:15.073208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:15.073178 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq"] Nov 04 00:35:15.079915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:15.079885 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b4376d-688dcbcd66-p98hq"] Nov 04 00:35:15.295055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:15.294963 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7d10bd-27ea-429e-8730-1b67bb6e7e46" path="/var/lib/kubelet/pods/da7d10bd-27ea-429e-8730-1b67bb6e7e46/volumes" Nov 04 00:35:15.736858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:15.736817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:16.525276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:16.525240 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:17.002860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:17.002813 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" podUID="2d4acee3-e8e1-4257-90bd-8aab3f96e757" containerName="payload-pre-processing" containerID="cri-o://2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3" gracePeriod=25 Nov 04 00:35:17.105602 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:35:17.105573 6208 logging.go:55] [core] [Channel #9886 SubChannel #9887]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.154:50052", ServerName: "10.133.0.154:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.154:50052: connect: connection refused" Nov 04 00:35:17.270978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:17.270943 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:17.285752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:17.285719 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:35:17.377622 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:17.377580 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4acee3-e8e1-4257-90bd-8aab3f96e757-kube-api-access-dz5n9\" (UniqueName: \"kubernetes.io/projected/2d4acee3-e8e1-4257-90bd-8aab3f96e757-kube-api-access-dz5n9\") pod \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\" (UID: \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\") " Nov 04 00:35:17.377820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:17.377681 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4acee3-e8e1-4257-90bd-8aab3f96e757-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4acee3-e8e1-4257-90bd-8aab3f96e757-plugins-config-volume\") pod \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\" (UID: \"2d4acee3-e8e1-4257-90bd-8aab3f96e757\") " Nov 04 00:35:17.378512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:17.378400 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4acee3-e8e1-4257-90bd-8aab3f96e757-plugins-config-volume" pod "2d4acee3-e8e1-4257-90bd-8aab3f96e757" (UID: "2d4acee3-e8e1-4257-90bd-8aab3f96e757"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:35:17.379999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:17.379966 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4acee3-e8e1-4257-90bd-8aab3f96e757-kube-api-access-dz5n9" pod "2d4acee3-e8e1-4257-90bd-8aab3f96e757" (UID: "2d4acee3-e8e1-4257-90bd-8aab3f96e757"). InnerVolumeSpecName "kube-api-access-dz5n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:35:17.479702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:17.479665 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz5n9\" (UniqueName: \"kubernetes.io/projected/2d4acee3-e8e1-4257-90bd-8aab3f96e757-kube-api-access-dz5n9\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:17.479702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:17.479696 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d4acee3-e8e1-4257-90bd-8aab3f96e757-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:18.068511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.068471 6208 generic.go:356] "Generic (PLEG): container finished" podID="2d4acee3-e8e1-4257-90bd-8aab3f96e757" containerID="2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3" exitCode=0 Nov 04 00:35:18.069013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.068518 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" Nov 04 00:35:18.069013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.068554 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" event={"ID":"2d4acee3-e8e1-4257-90bd-8aab3f96e757","Type":"ContainerDied","Data":"2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3"} Nov 04 00:35:18.069013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.068603 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" event={"ID":"2d4acee3-e8e1-4257-90bd-8aab3f96e757","Type":"ContainerDied","Data":"5d69872bddd2d0344531948410bc3f148e8053364cac6c45592e2b2c6ded1c8c"} Nov 04 00:35:18.069013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.068624 6208 scope.go:122] "RemoveContainer" containerID="2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3" Nov 04 00:35:18.076268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.076224 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:18.079718 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.079699 6208 scope.go:122] "RemoveContainer" containerID="2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3" Nov 04 00:35:18.080024 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:35:18.080004 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3\": container with ID starting with 2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3 not found: ID does not exist" containerID="2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3" Nov 04 00:35:18.080159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.080049 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3"} err="failed to get container status \"2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3\": rpc error: code = NotFound desc = could not find container \"2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3\": container with ID starting with 2e18cbbd7d4f1606c596d1b2e1437ebbcc10b486c3ca2223df7109db2fed0fb3 not found: ID does not exist" Nov 04 00:35:18.098211 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.098186 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79"] Nov 04 00:35:18.105810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.105771 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79"] Nov 04 00:35:18.106018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.105988 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4376d-7fd9885bf9-bjg79" podUID="2d4acee3-e8e1-4257-90bd-8aab3f96e757" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.154:50052\" within 1s: context deadline exceeded" Nov 04 00:35:18.106018 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:35:18.106013 6208 logging.go:55] [core] [Channel #9886 SubChannel #9887]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.154:50052", ServerName: "10.133.0.154:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.154:50052: operation was canceled" Nov 04 00:35:18.178342 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.178309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:18.553739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.553702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:18.921248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.921161 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l"] Nov 04 00:35:18.921427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.921386 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" containerID="cri-o://e6a8c63690629c9b2cc658c27bb8353142a0921805a8a2ffa301efbc0e769560" gracePeriod=30 Nov 04 00:35:18.951130 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:18.951097 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:19.294569 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:19.294533 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4acee3-e8e1-4257-90bd-8aab3f96e757" path="/var/lib/kubelet/pods/2d4acee3-e8e1-4257-90bd-8aab3f96e757/volumes" Nov 04 00:35:19.364867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:19.364830 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:19.721986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:19.721952 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:19.919285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:19.919251 6208 patch_prober.go:29] interesting pod/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:35:19.919452 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:19.919306 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:35:20.141687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:20.141594 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:20.950340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:20.950302 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:21.052702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:21.052668 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:21.534867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:21.534828 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:21.880714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:21.880629 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft"] Nov 04 00:35:21.880894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:21.880848 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" containerID="cri-o://1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25" gracePeriod=30 Nov 04 00:35:21.923521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:21.923488 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:22.023640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:22.023603 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:22.658269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:22.658233 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:22.769455 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:22.769422 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:23.954559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:23.954526 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:24.156845 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.156779 6208 generic.go:356] "Generic (PLEG): container finished" podID="418b6385-c834-4532-94d5-e65086de1352" containerID="e6a8c63690629c9b2cc658c27bb8353142a0921805a8a2ffa301efbc0e769560" exitCode=0 Nov 04 00:35:24.156845 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.156826 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" event={"ID":"418b6385-c834-4532-94d5-e65086de1352","Type":"ContainerDied","Data":"e6a8c63690629c9b2cc658c27bb8353142a0921805a8a2ffa301efbc0e769560"} Nov 04 00:35:24.157167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.156902 6208 scope.go:122] "RemoveContainer" containerID="6f2e4ee658d9b0c43dce210c1c2bc7954ce1b72b510cb3bb236892142a49605f" Nov 04 00:35:24.200929 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.200868 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:35:24.303135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303102 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-socket\") pod \"418b6385-c834-4532-94d5-e65086de1352\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " Nov 04 00:35:24.303324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303181 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-envoy\") pod \"418b6385-c834-4532-94d5-e65086de1352\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " Nov 04 00:35:24.303324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303205 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-kube-api-access-qlb7g\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-kube-api-access-qlb7g\") pod \"418b6385-c834-4532-94d5-e65086de1352\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " Nov 04 00:35:24.303324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303228 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-data\") pod \"418b6385-c834-4532-94d5-e65086de1352\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " Nov 04 00:35:24.303324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303246 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-certs\") pod \"418b6385-c834-4532-94d5-e65086de1352\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " Nov 04 00:35:24.303563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303354 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-credential-socket\") pod \"418b6385-c834-4532-94d5-e65086de1352\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " Nov 04 00:35:24.303563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303391 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-istio-token\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-istio-token\") pod \"418b6385-c834-4532-94d5-e65086de1352\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " Nov 04 00:35:24.303563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303426 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/418b6385-c834-4532-94d5-e65086de1352-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418b6385-c834-4532-94d5-e65086de1352-istio-podinfo\") pod \"418b6385-c834-4532-94d5-e65086de1352\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " Nov 04 00:35:24.303563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303452 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-socket" pod "418b6385-c834-4532-94d5-e65086de1352" (UID: "418b6385-c834-4532-94d5-e65086de1352"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:24.303563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303534 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418b6385-c834-4532-94d5-e65086de1352-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418b6385-c834-4532-94d5-e65086de1352-istiod-ca-cert\") pod \"418b6385-c834-4532-94d5-e65086de1352\" (UID: \"418b6385-c834-4532-94d5-e65086de1352\") " Nov 04 00:35:24.303891 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303556 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-data" pod "418b6385-c834-4532-94d5-e65086de1352" (UID: "418b6385-c834-4532-94d5-e65086de1352"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:24.303891 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303678 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-certs" pod "418b6385-c834-4532-94d5-e65086de1352" (UID: "418b6385-c834-4532-94d5-e65086de1352"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:24.304067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.303936 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418b6385-c834-4532-94d5-e65086de1352-istiod-ca-cert" pod "418b6385-c834-4532-94d5-e65086de1352" (UID: "418b6385-c834-4532-94d5-e65086de1352"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:35:24.304146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.304010 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-credential-socket" pod "418b6385-c834-4532-94d5-e65086de1352" (UID: "418b6385-c834-4532-94d5-e65086de1352"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:24.304213 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.304195 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:24.304275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.304219 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418b6385-c834-4532-94d5-e65086de1352-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:24.304275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.304234 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:24.304275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.304249 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:24.304275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.304263 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:24.305566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.305543 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-envoy" pod "418b6385-c834-4532-94d5-e65086de1352" (UID: "418b6385-c834-4532-94d5-e65086de1352"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:24.305675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.305576 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-kube-api-access-qlb7g" pod "418b6385-c834-4532-94d5-e65086de1352" (UID: "418b6385-c834-4532-94d5-e65086de1352"). InnerVolumeSpecName "kube-api-access-qlb7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:35:24.305675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.305600 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/418b6385-c834-4532-94d5-e65086de1352-istio-podinfo" pod "418b6385-c834-4532-94d5-e65086de1352" (UID: "418b6385-c834-4532-94d5-e65086de1352"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:35:24.305760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.305715 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-istio-token" pod "418b6385-c834-4532-94d5-e65086de1352" (UID: "418b6385-c834-4532-94d5-e65086de1352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:35:24.405886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.405842 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:24.405886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.405889 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418b6385-c834-4532-94d5-e65086de1352-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:24.406124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.405906 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418b6385-c834-4532-94d5-e65086de1352-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:24.406124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.405921 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlb7g\" (UniqueName: \"kubernetes.io/projected/418b6385-c834-4532-94d5-e65086de1352-kube-api-access-qlb7g\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:24.542739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.542701 6208 patch_prober.go:29] interesting pod/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:35:24.542916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.542769 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:35:24.767171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:24.767134 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:25.166300 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:25.166267 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" Nov 04 00:35:25.166300 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:25.166281 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l" event={"ID":"418b6385-c834-4532-94d5-e65086de1352","Type":"ContainerDied","Data":"0323e1cf51f030f0777ef03b92d2bf72f1481c0b5331bdf13305da05888fea43"} Nov 04 00:35:25.166887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:25.166336 6208 scope.go:122] "RemoveContainer" containerID="e6a8c63690629c9b2cc658c27bb8353142a0921805a8a2ffa301efbc0e769560" Nov 04 00:35:25.191458 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:25.191424 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l"] Nov 04 00:35:25.194266 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:25.194219 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-fa0d0a-openshift-default-6d485c86f-m6g9l"] Nov 04 00:35:25.293628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:25.293592 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="418b6385-c834-4532-94d5-e65086de1352" path="/var/lib/kubelet/pods/418b6385-c834-4532-94d5-e65086de1352/volumes" Nov 04 00:35:25.636010 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:25.635973 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:26.301303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:26.301274 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:26.414306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:26.414272 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:26.596295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:26.596203 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:27.148119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.148026 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:35:27.187666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.187627 6208 generic.go:356] "Generic (PLEG): container finished" podID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerID="1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25" exitCode=0 Nov 04 00:35:27.187844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.187765 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" event={"ID":"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb","Type":"ContainerDied","Data":"1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25"} Nov 04 00:35:27.187903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.187856 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" event={"ID":"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb","Type":"ContainerDied","Data":"8fc5b76f13a090b033d57eba0cd0b77859d588463891d51fb7754d56338de2f5"} Nov 04 00:35:27.187903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.187898 6208 scope.go:122] "RemoveContainer" containerID="1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25" Nov 04 00:35:27.191050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.188122 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft" Nov 04 00:35:27.205553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.205526 6208 scope.go:122] "RemoveContainer" containerID="84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271" Nov 04 00:35:27.216340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.216318 6208 scope.go:122] "RemoveContainer" containerID="1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25" Nov 04 00:35:27.216624 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:35:27.216604 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25\": container with ID starting with 1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25 not found: ID does not exist" containerID="1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25" Nov 04 00:35:27.216679 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.216634 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25"} err="failed to get container status \"1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25\": rpc error: code = NotFound desc = could not find container \"1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25\": container with ID starting with 1210e532a13cbdf453123e5cf67b77732c52ebbbaf5843b6e9a9ff5fca7a8b25 not found: ID does not exist" Nov 04 00:35:27.216679 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.216651 6208 scope.go:122] "RemoveContainer" containerID="84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271" Nov 04 00:35:27.216890 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:35:27.216872 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271\": container with ID starting with 84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271 not found: ID does not exist" containerID="84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271" Nov 04 00:35:27.216940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.216893 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271"} err="failed to get container status \"84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271\": rpc error: code = NotFound desc = could not find container \"84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271\": container with ID starting with 84b68b06a0583236c23c4dc1d81ce06f25b5569faf9ad4fb2f5fb1da612d9271 not found: ID does not exist" Nov 04 00:35:27.241309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241282 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-token\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-token\") pod \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " Nov 04 00:35:27.241418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241340 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-podinfo\") pod \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " Nov 04 00:35:27.241418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241387 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-envoy\") pod \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " Nov 04 00:35:27.241499 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241466 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istiod-ca-cert\") pod \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " Nov 04 00:35:27.241549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241523 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-socket\") pod \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " Nov 04 00:35:27.241651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241628 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-kube-api-access-vzwzh\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-kube-api-access-vzwzh\") pod \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " Nov 04 00:35:27.241795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241677 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-certs\") pod \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " Nov 04 00:35:27.241795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241748 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-data\") pod \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " Nov 04 00:35:27.241925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241813 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-credential-socket\") pod \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\" (UID: \"7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb\") " Nov 04 00:35:27.241925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241852 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-socket" pod "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" (UID: "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:27.241925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.241865 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istiod-ca-cert" pod "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" (UID: "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:35:27.242176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.242153 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-certs" pod "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" (UID: "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:27.242347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.242193 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-data" pod "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" (UID: "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:27.242347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.242297 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-credential-socket" pod "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" (UID: "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:27.242520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.242463 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:27.242520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.242484 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:27.242520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.242498 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:27.242520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.242512 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:27.242745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.242528 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:27.243697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.243671 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-envoy" pod "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" (UID: "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:27.243697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.243678 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-token" pod "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" (UID: "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:35:27.243821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.243701 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-podinfo" pod "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" (UID: "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:35:27.243969 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.243947 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-kube-api-access-vzwzh" pod "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" (UID: "7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb"). InnerVolumeSpecName "kube-api-access-vzwzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:35:27.343749 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.343716 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:27.343749 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.343747 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:27.344151 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.343758 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwzh\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-kube-api-access-vzwzh\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:27.344151 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.343769 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:27.508074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.508023 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft"] Nov 04 00:35:27.516789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.516759 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4376d-openshift-default-658df6c5fc-w9fft"] Nov 04 00:35:27.599685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:27.599652 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:28.346763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:28.346728 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:28.541316 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:28.541285 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:28.825858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:28.825823 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:28.936699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:28.936662 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:29.297459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:29.297421 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" path="/var/lib/kubelet/pods/7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb/volumes" Nov 04 00:35:29.354406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:29.354371 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:29.789104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:29.789073 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:30.806702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:30.806667 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:31.091633 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:31.091549 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:31.948088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:31.948056 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:32.755337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:32.755300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:33.574408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:33.574369 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:34.206625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:34.206587 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:34.566228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:34.566140 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:35.485440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:35.485404 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:35.591666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:35.591615 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:35.692332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:35.692297 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:35.707075 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:35:35.707022 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:35:36.101803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:36.101767 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:36.345970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:36.345936 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:36.465068 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:36.465022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:37.227779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:37.227736 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:37.349485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:37.349447 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:38.022130 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:38.022084 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:38.754170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:38.754135 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:38.890409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:38.890377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:39.093931 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:39.093843 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:39.372253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:39.372162 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:39.723729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:39.723695 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs"] Nov 04 00:35:39.723972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:39.723950 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" containerID="cri-o://93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc" gracePeriod=30 Nov 04 00:35:39.732153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:39.732129 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:39.941185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:39.941153 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:40.496623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:40.496582 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:40.870263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:40.870176 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:41.697407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:41.697370 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:42.644123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:42.644087 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:43.531962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:43.531932 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:44.436475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:44.436426 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:44.534933 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:44.534891 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:44.732605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:44.732571 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:44.895165 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:44.895129 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 00:35:44.895375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:44.895209 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 00:35:45.003721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.003694 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 04 00:35:45.147976 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.147942 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-data\") pod \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " Nov 04 00:35:45.147976 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.147982 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-kube-api-access-9xgnr\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-kube-api-access-9xgnr\") pod \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " Nov 04 00:35:45.148260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148048 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-podinfo\") pod \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " Nov 04 00:35:45.148260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148084 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-certs\") pod \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " Nov 04 00:35:45.148260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148129 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istiod-ca-cert\") pod \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " Nov 04 00:35:45.148260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148189 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-socket\") pod \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " Nov 04 00:35:45.148260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148241 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-credential-socket\") pod \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " Nov 04 00:35:45.148527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148326 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-data" pod "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" (UID: "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:45.148527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148364 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-token\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-token\") pod \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " Nov 04 00:35:45.148527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148415 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-envoy\") pod \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\" (UID: \"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79\") " Nov 04 00:35:45.148706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148624 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-certs" pod "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" (UID: "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:45.148706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148644 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-credential-socket" pod "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" (UID: "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:45.148798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148713 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-socket" pod "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" (UID: "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:45.148798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148746 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istiod-ca-cert" pod "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" (UID: "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:35:45.148969 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148949 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:45.148969 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148969 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:45.148969 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148979 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:45.149199 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148987 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:45.149199 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.148999 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:45.150442 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.150414 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-podinfo" pod "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" (UID: "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:35:45.150862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.150737 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-envoy" pod "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" (UID: "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:35:45.150862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.150828 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-kube-api-access-9xgnr" pod "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" (UID: "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79"). InnerVolumeSpecName "kube-api-access-9xgnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:35:45.150862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.150849 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-token" pod "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" (UID: "b29233c7-5e58-4ed7-b12c-6aa0ffe49e79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:35:45.249977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.249891 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xgnr\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-kube-api-access-9xgnr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:45.249977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.249923 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:45.249977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.249933 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:45.249977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.249944 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:45.312029 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.311997 6208 generic.go:356] "Generic (PLEG): container finished" podID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerID="93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc" exitCode=0 Nov 04 00:35:45.312221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.312078 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" Nov 04 00:35:45.312221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.312082 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" event={"ID":"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79","Type":"ContainerDied","Data":"93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc"} Nov 04 00:35:45.312221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.312144 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs" event={"ID":"b29233c7-5e58-4ed7-b12c-6aa0ffe49e79","Type":"ContainerDied","Data":"600d8209a9022f520bef6ed2edb8c6ef5432acb9d1327edad5270e2493a6614a"} Nov 04 00:35:45.312221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.312176 6208 scope.go:122] "RemoveContainer" containerID="93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc" Nov 04 00:35:45.322862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.322842 6208 scope.go:122] "RemoveContainer" containerID="93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc" Nov 04 00:35:45.323177 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:35:45.323156 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc\": container with ID starting with 93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc not found: ID does not exist" containerID="93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc" Nov 04 00:35:45.323229 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.323185 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc"} err="failed to get container status \"93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc\": rpc error: code = NotFound desc = could not find container \"93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc\": container with ID starting with 93d7b54650ea8609aa7e7787b9dbd8c082eeb504705c1396df6b0ea2c2d548bc not found: ID does not exist" Nov 04 00:35:45.387796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.387752 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs"] Nov 04 00:35:45.401865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.401825 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:45.419378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:45.419345 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-58b75d-openshift-default-5d5cbfc5f7-ncwvs"] Nov 04 00:35:46.344893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:46.344850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:46.762269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:46.762235 6208 scope.go:122] "RemoveContainer" containerID="4c0ace1f1efc9c0e7c762eeb379142967c6489cf117281650e998f8d58439623" Nov 04 00:35:47.013394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:47.013308 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:47.113147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:47.113116 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:47.294836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:47.294757 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" path="/var/lib/kubelet/pods/b29233c7-5e58-4ed7-b12c-6aa0ffe49e79/volumes" Nov 04 00:35:48.048764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:48.048727 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:48.237088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:48.237048 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:48.787670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:48.787635 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:49.535315 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:49.535276 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:50.290095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:50.290060 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:50.615054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:50.614921 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:51.362163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:51.362129 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:51.742255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:51.742214 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:51.844969 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:51.844927 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:52.176729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:52.176647 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:52.277176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:52.277138 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:53.197825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:53.197789 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:53.714952 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:53.714923 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:53.838704 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:53.838672 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:53.917594 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:53.917562 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p"] Nov 04 00:35:53.917786 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:53.917765 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" podUID="6fa02593-a635-47a7-a21f-9d67b1e8018e" containerName="maas-api" containerID="cri-o://ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea" gracePeriod=30 Nov 04 00:35:53.975874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:53.975799 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:54.265163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.265128 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 04 00:35:54.285796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.285759 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:54.374355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.374320 6208 generic.go:356] "Generic (PLEG): container finished" podID="6fa02593-a635-47a7-a21f-9d67b1e8018e" containerID="ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea" exitCode=0 Nov 04 00:35:54.374625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.374366 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" Nov 04 00:35:54.374625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.374404 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" event={"ID":"6fa02593-a635-47a7-a21f-9d67b1e8018e","Type":"ContainerDied","Data":"ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea"} Nov 04 00:35:54.374625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.374459 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p" event={"ID":"6fa02593-a635-47a7-a21f-9d67b1e8018e","Type":"ContainerDied","Data":"bd39b99e09491a15d3e0bf6cb4660095cd570069ba50ecabb535003077e94051"} Nov 04 00:35:54.374625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.374487 6208 scope.go:122] "RemoveContainer" containerID="ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea" Nov 04 00:35:54.386625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.386603 6208 scope.go:122] "RemoveContainer" containerID="ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea" Nov 04 00:35:54.386929 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:35:54.386901 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea\": container with ID starting with ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea not found: ID does not exist" containerID="ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea" Nov 04 00:35:54.386983 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.386940 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea"} err="failed to get container status \"ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea\": rpc error: code = NotFound desc = could not find container \"ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea\": container with ID starting with ac880dbd50c3ed0f3a9053ce0ec5c043467d0c79ffad9d13eb01a7fb3a98c1ea not found: ID does not exist" Nov 04 00:35:54.398389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.398369 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa02593-a635-47a7-a21f-9d67b1e8018e-kube-api-access-lk47l\" (UniqueName: \"kubernetes.io/projected/6fa02593-a635-47a7-a21f-9d67b1e8018e-kube-api-access-lk47l\") pod \"6fa02593-a635-47a7-a21f-9d67b1e8018e\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " Nov 04 00:35:54.398509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.398498 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-metrics-certs\") pod \"6fa02593-a635-47a7-a21f-9d67b1e8018e\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " Nov 04 00:35:54.398555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.398546 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-maas-api-tls\") pod \"6fa02593-a635-47a7-a21f-9d67b1e8018e\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " Nov 04 00:35:54.398673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.398649 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa02593-a635-47a7-a21f-9d67b1e8018e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa02593-a635-47a7-a21f-9d67b1e8018e-openshift-service-ca\") pod \"6fa02593-a635-47a7-a21f-9d67b1e8018e\" (UID: \"6fa02593-a635-47a7-a21f-9d67b1e8018e\") " Nov 04 00:35:54.420059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.416395 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa02593-a635-47a7-a21f-9d67b1e8018e-openshift-service-ca" pod "6fa02593-a635-47a7-a21f-9d67b1e8018e" (UID: "6fa02593-a635-47a7-a21f-9d67b1e8018e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:35:54.420059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.419737 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa02593-a635-47a7-a21f-9d67b1e8018e-kube-api-access-lk47l" pod "6fa02593-a635-47a7-a21f-9d67b1e8018e" (UID: "6fa02593-a635-47a7-a21f-9d67b1e8018e"). InnerVolumeSpecName "kube-api-access-lk47l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:35:54.431701 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.429740 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-maas-api-tls" pod "6fa02593-a635-47a7-a21f-9d67b1e8018e" (UID: "6fa02593-a635-47a7-a21f-9d67b1e8018e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:35:54.431701 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.431275 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-metrics-certs" pod "6fa02593-a635-47a7-a21f-9d67b1e8018e" (UID: "6fa02593-a635-47a7-a21f-9d67b1e8018e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:35:54.500449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.500370 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:54.500449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.500396 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa02593-a635-47a7-a21f-9d67b1e8018e-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:54.500449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.500409 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa02593-a635-47a7-a21f-9d67b1e8018e-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:54.500449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.500418 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk47l\" (UniqueName: \"kubernetes.io/projected/6fa02593-a635-47a7-a21f-9d67b1e8018e-kube-api-access-lk47l\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:35:54.699132 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.699099 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p"] Nov 04 00:35:54.704276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:54.704241 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8093f5-756fb84b7-d5p9p"] Nov 04 00:35:55.121667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:55.121612 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:55.294784 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:55.294747 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa02593-a635-47a7-a21f-9d67b1e8018e" path="/var/lib/kubelet/pods/6fa02593-a635-47a7-a21f-9d67b1e8018e/volumes" Nov 04 00:35:55.710022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:55.709963 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:56.053706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:56.053616 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:56.966863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:56.966828 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:57.896827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:57.896791 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:58.096518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:58.096480 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:35:59.264124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:35:59.264089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:00.341514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:00.341478 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:01.240295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:01.240245 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:02.018425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:02.018387 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:02.601464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:02.601435 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:02.804680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:02.804643 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:03.029527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:03.029490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:03.134755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:03.134725 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:03.428785 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:03.428690 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:03.537565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:03.537525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:03.907250 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:03.907214 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:03.935763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:03.935722 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj"] Nov 04 00:36:03.936043 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:03.935997 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" containerID="cri-o://7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746" gracePeriod=30 Nov 04 00:36:04.003386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:04.003348 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:04.438238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:04.438199 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:04.541914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:04.541871 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:04.838138 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:04.838093 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:05.647924 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:05.647887 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:05.736266 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:36:05.736229 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:36:05.945418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:05.945328 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:06.780341 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:06.780300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:07.547300 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:07.547254 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:08.490574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:08.490534 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:08.776677 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:08.776588 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:08.874954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:08.874919 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:09.185661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.185567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:09.220060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.219997 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 04 00:36:09.332236 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.332156 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-podinfo\") pod \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " Nov 04 00:36:09.332236 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.332216 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istiod-ca-cert\") pod \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " Nov 04 00:36:09.332473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.332277 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-credential-socket\") pod \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " Nov 04 00:36:09.332473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.332329 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-socket\") pod \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " Nov 04 00:36:09.332473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.332359 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-token\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-token\") pod \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " Nov 04 00:36:09.332473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.332393 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-certs\") pod \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " Nov 04 00:36:09.332640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.332481 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-data\") pod \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " Nov 04 00:36:09.332640 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.332615 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-envoy\") pod \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " Nov 04 00:36:09.332715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.332652 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-kube-api-access-x4r2m\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-kube-api-access-x4r2m\") pod \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\" (UID: \"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f\") " Nov 04 00:36:09.333808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.333776 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-socket" pod "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" (UID: "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:09.334410 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.334384 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istiod-ca-cert" pod "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" (UID: "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:36:09.335194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.334678 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-credential-socket" pod "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" (UID: "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:09.335552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.334814 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-certs" pod "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" (UID: "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:09.335675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.335080 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-data" pod "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" (UID: "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:09.339234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.339209 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-kube-api-access-x4r2m" pod "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" (UID: "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f"). InnerVolumeSpecName "kube-api-access-x4r2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:09.339467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.339402 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-token" pod "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" (UID: "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:09.339467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.339388 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-podinfo" pod "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" (UID: "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:36:09.339467 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.339395 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-envoy" pod "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" (UID: "eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:09.396193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.396151 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd"] Nov 04 00:36:09.414961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.414926 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9"] Nov 04 00:36:09.434500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.434470 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:09.434500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.434496 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:09.434702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.434509 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:09.434702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.434519 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:09.434702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.434528 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:09.434702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.434536 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:09.434702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.434545 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4r2m\" (UniqueName: \"kubernetes.io/projected/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-kube-api-access-x4r2m\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:09.434702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.434554 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:09.434702 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.434562 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:09.465867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.465830 6208 generic.go:356] "Generic (PLEG): container finished" podID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerID="7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746" exitCode=0 Nov 04 00:36:09.466090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.465907 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" Nov 04 00:36:09.466090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.465915 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" event={"ID":"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f","Type":"ContainerDied","Data":"7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746"} Nov 04 00:36:09.466090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.465979 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj" event={"ID":"eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f","Type":"ContainerDied","Data":"8a8bc403088ce68d24a4df23c589cd2d7f3cddbb10dda67d3ad5b5dc318d5216"} Nov 04 00:36:09.466090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.466002 6208 scope.go:122] "RemoveContainer" containerID="7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746" Nov 04 00:36:09.477656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.477639 6208 scope.go:122] "RemoveContainer" containerID="16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6" Nov 04 00:36:09.487829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.487810 6208 scope.go:122] "RemoveContainer" containerID="7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746" Nov 04 00:36:09.490195 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:36:09.490166 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746\": container with ID starting with 7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746 not found: ID does not exist" containerID="7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746" Nov 04 00:36:09.490304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.490207 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746"} err="failed to get container status \"7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746\": rpc error: code = NotFound desc = could not find container \"7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746\": container with ID starting with 7871f963e5968673c55454195fdcd903b8cc604dd68c4fb1a4dc080280acf746 not found: ID does not exist" Nov 04 00:36:09.490304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.490230 6208 scope.go:122] "RemoveContainer" containerID="16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6" Nov 04 00:36:09.490539 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:36:09.490515 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6\": container with ID starting with 16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6 not found: ID does not exist" containerID="16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6" Nov 04 00:36:09.490600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.490546 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6"} err="failed to get container status \"16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6\": rpc error: code = NotFound desc = could not find container \"16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6\": container with ID starting with 16b9fb49accc78771c092e77ef66e93bb8efd88d021d834a42a2f501fbd7f7b6 not found: ID does not exist" Nov 04 00:36:09.491242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.491223 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj"] Nov 04 00:36:09.500540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.500507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:09.502905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.502877 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8093f5-openshift-default-77b94cf8cf-m2bjj"] Nov 04 00:36:09.616920 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.616824 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:09.720767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:09.720707 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:10.080146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:10.080109 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:10.320075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:10.320013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:11.124662 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:11.124622 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:11.294291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:11.294258 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" path="/var/lib/kubelet/pods/eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f/volumes" Nov 04 00:36:11.996230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:11.996195 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:12.639497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:12.639463 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:13.231104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:13.231024 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:13.890266 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:13.890226 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:14.170900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:14.170801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:14.978580 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:14.978534 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:15.392947 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:15.392858 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:15.506114 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:15.506072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:15.892240 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:15.892205 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:16.898301 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:16.898262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:17.016685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:17.016646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:17.835625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:17.835569 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:18.238727 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:18.238694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:18.576725 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:18.576635 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:18.925547 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:18.925455 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:19.660553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:19.660501 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:20.349533 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:20.349497 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:20.575140 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:20.575101 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:20.701005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:20.700915 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:20.914718 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:20.914685 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:21.302510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:21.302470 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:21.664679 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:21.664592 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:22.552202 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:22.552165 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:23.351837 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:23.351801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:24.111918 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.111878 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:24.305601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.305565 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:24.420425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.420342 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" containerID="cri-o://fd51b633090c1c0ba023535c2b35ab398d784609cb403c6298a33e91691e2723" gracePeriod=45 Nov 04 00:36:24.425408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.425377 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:24.443502 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.443440 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" containerID="cri-o://dfde2ba6a713e9f98db94b7c83af1ad098f927a0d6c2fd055d31069d435a75ac" gracePeriod=45 Nov 04 00:36:24.563536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.563510 6208 generic.go:356] "Generic (PLEG): container finished" podID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerID="fd51b633090c1c0ba023535c2b35ab398d784609cb403c6298a33e91691e2723" exitCode=0 Nov 04 00:36:24.563662 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.563574 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" event={"ID":"e200da9d-3c7d-4904-9d6d-88badac0a185","Type":"ContainerDied","Data":"fd51b633090c1c0ba023535c2b35ab398d784609cb403c6298a33e91691e2723"} Nov 04 00:36:24.565535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.565509 6208 generic.go:356] "Generic (PLEG): container finished" podID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerID="dfde2ba6a713e9f98db94b7c83af1ad098f927a0d6c2fd055d31069d435a75ac" exitCode=0 Nov 04 00:36:24.565662 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.565561 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" event={"ID":"a66e1c08-6769-477b-bb4a-1970e09e6627","Type":"ContainerDied","Data":"dfde2ba6a713e9f98db94b7c83af1ad098f927a0d6c2fd055d31069d435a75ac"} Nov 04 00:36:24.720181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.716379 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 04 00:36:24.759541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.759509 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 04 00:36:24.848142 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848102 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e200da9d-3c7d-4904-9d6d-88badac0a185-tls-certs\" (UniqueName: \"kubernetes.io/secret/e200da9d-3c7d-4904-9d6d-88badac0a185-tls-certs\") pod \"e200da9d-3c7d-4904-9d6d-88badac0a185\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " Nov 04 00:36:24.848142 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848151 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-tmp-dir\") pod \"e200da9d-3c7d-4904-9d6d-88badac0a185\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " Nov 04 00:36:24.848418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848245 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-model-cache\") pod \"e200da9d-3c7d-4904-9d6d-88badac0a185\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " Nov 04 00:36:24.848418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848289 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-home\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-home\") pod \"e200da9d-3c7d-4904-9d6d-88badac0a185\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " Nov 04 00:36:24.848418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848314 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-dshm\") pod \"e200da9d-3c7d-4904-9d6d-88badac0a185\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " Nov 04 00:36:24.848418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848357 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e200da9d-3c7d-4904-9d6d-88badac0a185-kube-api-access-6x5k5\" (UniqueName: \"kubernetes.io/projected/e200da9d-3c7d-4904-9d6d-88badac0a185-kube-api-access-6x5k5\") pod \"e200da9d-3c7d-4904-9d6d-88badac0a185\" (UID: \"e200da9d-3c7d-4904-9d6d-88badac0a185\") " Nov 04 00:36:24.848892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848716 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-model-cache" pod "e200da9d-3c7d-4904-9d6d-88badac0a185" (UID: "e200da9d-3c7d-4904-9d6d-88badac0a185"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:24.848892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848727 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-home" pod "e200da9d-3c7d-4904-9d6d-88badac0a185" (UID: "e200da9d-3c7d-4904-9d6d-88badac0a185"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:24.848892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848851 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-tmp-dir" pod "e200da9d-3c7d-4904-9d6d-88badac0a185" (UID: "e200da9d-3c7d-4904-9d6d-88badac0a185"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:24.849168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848928 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:24.849168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.848949 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:24.850646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.850626 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-dshm" pod "e200da9d-3c7d-4904-9d6d-88badac0a185" (UID: "e200da9d-3c7d-4904-9d6d-88badac0a185"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:24.850719 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.850688 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e200da9d-3c7d-4904-9d6d-88badac0a185-tls-certs" pod "e200da9d-3c7d-4904-9d6d-88badac0a185" (UID: "e200da9d-3c7d-4904-9d6d-88badac0a185"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:36:24.851081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.851059 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e200da9d-3c7d-4904-9d6d-88badac0a185-kube-api-access-6x5k5" pod "e200da9d-3c7d-4904-9d6d-88badac0a185" (UID: "e200da9d-3c7d-4904-9d6d-88badac0a185"). InnerVolumeSpecName "kube-api-access-6x5k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:24.949850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.949753 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-home\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-home\") pod \"a66e1c08-6769-477b-bb4a-1970e09e6627\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " Nov 04 00:36:24.949850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.949811 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-tmp-dir\") pod \"a66e1c08-6769-477b-bb4a-1970e09e6627\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " Nov 04 00:36:24.950128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.949852 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a66e1c08-6769-477b-bb4a-1970e09e6627-tls-certs\" (UniqueName: \"kubernetes.io/secret/a66e1c08-6769-477b-bb4a-1970e09e6627-tls-certs\") pod \"a66e1c08-6769-477b-bb4a-1970e09e6627\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " Nov 04 00:36:24.950128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.949928 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-dshm\") pod \"a66e1c08-6769-477b-bb4a-1970e09e6627\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " Nov 04 00:36:24.950128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.949985 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a66e1c08-6769-477b-bb4a-1970e09e6627-kube-api-access-hfs2q\" (UniqueName: \"kubernetes.io/projected/a66e1c08-6769-477b-bb4a-1970e09e6627-kube-api-access-hfs2q\") pod \"a66e1c08-6769-477b-bb4a-1970e09e6627\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " Nov 04 00:36:24.950291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950139 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-model-cache\") pod \"a66e1c08-6769-477b-bb4a-1970e09e6627\" (UID: \"a66e1c08-6769-477b-bb4a-1970e09e6627\") " Nov 04 00:36:24.950291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950178 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-home" pod "a66e1c08-6769-477b-bb4a-1970e09e6627" (UID: "a66e1c08-6769-477b-bb4a-1970e09e6627"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:24.950291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950191 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-tmp-dir" pod "a66e1c08-6769-477b-bb4a-1970e09e6627" (UID: "a66e1c08-6769-477b-bb4a-1970e09e6627"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:24.950463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950430 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-model-cache" pod "a66e1c08-6769-477b-bb4a-1970e09e6627" (UID: "a66e1c08-6769-477b-bb4a-1970e09e6627"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:24.950632 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950598 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:24.950632 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950620 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x5k5\" (UniqueName: \"kubernetes.io/projected/e200da9d-3c7d-4904-9d6d-88badac0a185-kube-api-access-6x5k5\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:24.950804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950636 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:24.950804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950651 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e200da9d-3c7d-4904-9d6d-88badac0a185-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:24.950804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950684 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200da9d-3c7d-4904-9d6d-88badac0a185-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:24.950804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950697 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:24.950804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.950711 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:24.952202 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.952182 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66e1c08-6769-477b-bb4a-1970e09e6627-tls-certs" pod "a66e1c08-6769-477b-bb4a-1970e09e6627" (UID: "a66e1c08-6769-477b-bb4a-1970e09e6627"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:36:24.952272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.952186 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-dshm" pod "a66e1c08-6769-477b-bb4a-1970e09e6627" (UID: "a66e1c08-6769-477b-bb4a-1970e09e6627"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:24.952272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:24.952218 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66e1c08-6769-477b-bb4a-1970e09e6627-kube-api-access-hfs2q" pod "a66e1c08-6769-477b-bb4a-1970e09e6627" (UID: "a66e1c08-6769-477b-bb4a-1970e09e6627"). InnerVolumeSpecName "kube-api-access-hfs2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:25.051655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.051610 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66e1c08-6769-477b-bb4a-1970e09e6627-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:25.051655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.051648 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66e1c08-6769-477b-bb4a-1970e09e6627-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:25.051891 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.051665 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfs2q\" (UniqueName: \"kubernetes.io/projected/a66e1c08-6769-477b-bb4a-1970e09e6627-kube-api-access-hfs2q\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:25.276321 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.276288 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:25.427053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.416967 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:25.573838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.573754 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" Nov 04 00:36:25.573838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.573775 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9" event={"ID":"a66e1c08-6769-477b-bb4a-1970e09e6627","Type":"ContainerDied","Data":"f417c481bc6ba610ce9bd9e9355000b150da3342196515ec0d4ca98677eeaa6c"} Nov 04 00:36:25.574121 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.573859 6208 scope.go:122] "RemoveContainer" containerID="dfde2ba6a713e9f98db94b7c83af1ad098f927a0d6c2fd055d31069d435a75ac" Nov 04 00:36:25.576610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.576007 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" Nov 04 00:36:25.576610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.576026 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd" event={"ID":"e200da9d-3c7d-4904-9d6d-88badac0a185","Type":"ContainerDied","Data":"f5f9a9439bba2fe23f1b508766d25295c044cd801c0a808ca53e3e3fc671d934"} Nov 04 00:36:25.586720 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.586688 6208 scope.go:122] "RemoveContainer" containerID="fd51b633090c1c0ba023535c2b35ab398d784609cb403c6298a33e91691e2723" Nov 04 00:36:25.598537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.598511 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9"] Nov 04 00:36:25.602837 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.602811 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/simulator-w2-5738a9-kserve-659975c56b-hxqc9"] Nov 04 00:36:25.619173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.619140 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd"] Nov 04 00:36:25.621344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.621320 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5738a9/distinct-w2-5738a9-kserve-59c866dd6-l4zxd"] Nov 04 00:36:25.652096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.652014 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:25.995305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.995271 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr"] Nov 04 00:36:25.995535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:25.995509 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" containerID="cri-o://f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45" gracePeriod=30 Nov 04 00:36:26.031525 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:26.031482 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:26.270748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:26.270657 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:26.370092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:26.370048 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:26.647320 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:26.647223 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:26.861123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:26.861076 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:27.114828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:27.114783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:27.297530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:27.297488 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" path="/var/lib/kubelet/pods/a66e1c08-6769-477b-bb4a-1970e09e6627/volumes" Nov 04 00:36:27.298194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:27.298172 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" path="/var/lib/kubelet/pods/e200da9d-3c7d-4904-9d6d-88badac0a185/volumes" Nov 04 00:36:27.547736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:27.547687 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:28.468827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:28.468792 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:29.231681 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:29.231640 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:30.031759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:30.031715 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:30.812339 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:30.812302 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:31.253304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.253271 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:36:31.322627 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.322584 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b9470d-1121-4176-be18-bd5348ae17db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b9470d-1121-4176-be18-bd5348ae17db-istiod-ca-cert\") pod \"77b9470d-1121-4176-be18-bd5348ae17db\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " Nov 04 00:36:31.322807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.322633 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-data\") pod \"77b9470d-1121-4176-be18-bd5348ae17db\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " Nov 04 00:36:31.322807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.322681 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-socket\") pod \"77b9470d-1121-4176-be18-bd5348ae17db\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " Nov 04 00:36:31.322807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.322740 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-istio-token\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-istio-token\") pod \"77b9470d-1121-4176-be18-bd5348ae17db\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " Nov 04 00:36:31.322807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.322769 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77b9470d-1121-4176-be18-bd5348ae17db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b9470d-1121-4176-be18-bd5348ae17db-istio-podinfo\") pod \"77b9470d-1121-4176-be18-bd5348ae17db\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " Nov 04 00:36:31.323067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.322837 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-kube-api-access-2jx4s\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-kube-api-access-2jx4s\") pod \"77b9470d-1121-4176-be18-bd5348ae17db\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " Nov 04 00:36:31.323067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.322927 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-envoy\") pod \"77b9470d-1121-4176-be18-bd5348ae17db\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " Nov 04 00:36:31.323067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.322975 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-data" pod "77b9470d-1121-4176-be18-bd5348ae17db" (UID: "77b9470d-1121-4176-be18-bd5348ae17db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:31.323067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.322994 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-certs\") pod \"77b9470d-1121-4176-be18-bd5348ae17db\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " Nov 04 00:36:31.323301 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.323064 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-credential-socket\") pod \"77b9470d-1121-4176-be18-bd5348ae17db\" (UID: \"77b9470d-1121-4176-be18-bd5348ae17db\") " Nov 04 00:36:31.323301 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.323078 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b9470d-1121-4176-be18-bd5348ae17db-istiod-ca-cert" pod "77b9470d-1121-4176-be18-bd5348ae17db" (UID: "77b9470d-1121-4176-be18-bd5348ae17db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:36:31.323429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.323293 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-certs" pod "77b9470d-1121-4176-be18-bd5348ae17db" (UID: "77b9470d-1121-4176-be18-bd5348ae17db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:31.323615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.323588 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-socket" pod "77b9470d-1121-4176-be18-bd5348ae17db" (UID: "77b9470d-1121-4176-be18-bd5348ae17db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:31.323820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.323798 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-credential-socket" pod "77b9470d-1121-4176-be18-bd5348ae17db" (UID: "77b9470d-1121-4176-be18-bd5348ae17db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:31.325524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.325491 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77b9470d-1121-4176-be18-bd5348ae17db-istio-podinfo" pod "77b9470d-1121-4176-be18-bd5348ae17db" (UID: "77b9470d-1121-4176-be18-bd5348ae17db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:36:31.325785 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.325765 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:31.325880 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.325791 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:31.325880 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.325815 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b9470d-1121-4176-be18-bd5348ae17db-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:31.325880 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.325764 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-envoy" pod "77b9470d-1121-4176-be18-bd5348ae17db" (UID: "77b9470d-1121-4176-be18-bd5348ae17db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:31.325880 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.325856 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-kube-api-access-2jx4s" pod "77b9470d-1121-4176-be18-bd5348ae17db" (UID: "77b9470d-1121-4176-be18-bd5348ae17db"). InnerVolumeSpecName "kube-api-access-2jx4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:31.326154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.326132 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-istio-token" pod "77b9470d-1121-4176-be18-bd5348ae17db" (UID: "77b9470d-1121-4176-be18-bd5348ae17db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:31.426928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.426832 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jx4s\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-kube-api-access-2jx4s\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:31.426928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.426870 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:31.426928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.426884 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:31.426928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.426899 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b9470d-1121-4176-be18-bd5348ae17db-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:31.426928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.426912 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b9470d-1121-4176-be18-bd5348ae17db-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:31.426928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.426925 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b9470d-1121-4176-be18-bd5348ae17db-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:31.631273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.631230 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:31.632873 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.632843 6208 generic.go:356] "Generic (PLEG): container finished" podID="77b9470d-1121-4176-be18-bd5348ae17db" containerID="f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45" exitCode=0 Nov 04 00:36:31.633013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.632903 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" Nov 04 00:36:31.633013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.632920 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" event={"ID":"77b9470d-1121-4176-be18-bd5348ae17db","Type":"ContainerDied","Data":"f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45"} Nov 04 00:36:31.633013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.632976 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr" event={"ID":"77b9470d-1121-4176-be18-bd5348ae17db","Type":"ContainerDied","Data":"8a4825d33d1f7f56ad5fd30f8e085428c04e657140572b06e1f99a918330b42e"} Nov 04 00:36:31.633013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.633005 6208 scope.go:122] "RemoveContainer" containerID="f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45" Nov 04 00:36:31.644543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.644524 6208 scope.go:122] "RemoveContainer" containerID="72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7" Nov 04 00:36:31.657264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.657240 6208 scope.go:122] "RemoveContainer" containerID="f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45" Nov 04 00:36:31.664082 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:36:31.657867 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45\": container with ID starting with f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45 not found: ID does not exist" containerID="f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45" Nov 04 00:36:31.664082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.657934 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45"} err="failed to get container status \"f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45\": rpc error: code = NotFound desc = could not find container \"f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45\": container with ID starting with f9d7de25420c0482511a68a77acaf9169c7ef7a7f0f1e6f072c4df28bbcbde45 not found: ID does not exist" Nov 04 00:36:31.664082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.657959 6208 scope.go:122] "RemoveContainer" containerID="72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7" Nov 04 00:36:31.664082 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:36:31.658635 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7\": container with ID starting with 72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7 not found: ID does not exist" containerID="72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7" Nov 04 00:36:31.664082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.658664 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7"} err="failed to get container status \"72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7\": rpc error: code = NotFound desc = could not find container \"72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7\": container with ID starting with 72f6ac90dfe2ed1268929cbefb3d57f4e0760272ba4e0ae8e7244b3aca89c4b7 not found: ID does not exist" Nov 04 00:36:31.664082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.661923 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr"] Nov 04 00:36:31.667174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.667150 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8895cb-openshift-default-b946df8cc-k55zr"] Nov 04 00:36:31.731966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:31.731880 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:32.420942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:32.420908 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt"] Nov 04 00:36:32.447181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:32.447142 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:32.545109 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:32.545078 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:33.293823 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:33.293791 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b9470d-1121-4176-be18-bd5348ae17db" path="/var/lib/kubelet/pods/77b9470d-1121-4176-be18-bd5348ae17db/volumes" Nov 04 00:36:33.463740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:33.463702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:34.321354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:34.321312 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:35.129691 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:35.129644 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:35.596803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:35.596762 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:35.695942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:35.695903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:35.706322 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:36:35.706285 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:36:35.923798 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:35.923697 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:36.559775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:36.559741 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:36.733276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:36.733240 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95"] Nov 04 00:36:36.750982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:36.750930 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr"] Nov 04 00:36:36.804005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:36.803971 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:36.902910 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:36.902819 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:37.370810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.370772 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:37.421486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.421434 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" podUID="62966127-1cff-4a1a-b9d5-16de42660432" containerName="payload-processing" containerID="cri-o://0671c050e8723da33a720d98b163d06b30d579e6247146a63cf18a0f209ea583" gracePeriod=25 Nov 04 00:36:37.689838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.689798 6208 generic.go:356] "Generic (PLEG): container finished" podID="62966127-1cff-4a1a-b9d5-16de42660432" containerID="0671c050e8723da33a720d98b163d06b30d579e6247146a63cf18a0f209ea583" exitCode=0 Nov 04 00:36:37.690265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.689872 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" event={"ID":"62966127-1cff-4a1a-b9d5-16de42660432","Type":"ContainerDied","Data":"0671c050e8723da33a720d98b163d06b30d579e6247146a63cf18a0f209ea583"} Nov 04 00:36:37.690265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.689938 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" event={"ID":"62966127-1cff-4a1a-b9d5-16de42660432","Type":"ContainerDied","Data":"b6814a2a6d494faea6a82058c973d506d7d715ad62715fb17055c05accf6f352"} Nov 04 00:36:37.690265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.689968 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6814a2a6d494faea6a82058c973d506d7d715ad62715fb17055c05accf6f352" Nov 04 00:36:37.705094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.705069 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:36:37.740141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.740107 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:37.846081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.846023 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62966127-1cff-4a1a-b9d5-16de42660432-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62966127-1cff-4a1a-b9d5-16de42660432-plugins-config-volume\") pod \"62966127-1cff-4a1a-b9d5-16de42660432\" (UID: \"62966127-1cff-4a1a-b9d5-16de42660432\") " Nov 04 00:36:37.846302 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.846136 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62966127-1cff-4a1a-b9d5-16de42660432-kube-api-access-pw6br\" (UniqueName: \"kubernetes.io/projected/62966127-1cff-4a1a-b9d5-16de42660432-kube-api-access-pw6br\") pod \"62966127-1cff-4a1a-b9d5-16de42660432\" (UID: \"62966127-1cff-4a1a-b9d5-16de42660432\") " Nov 04 00:36:37.846503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.846479 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62966127-1cff-4a1a-b9d5-16de42660432-plugins-config-volume" pod "62966127-1cff-4a1a-b9d5-16de42660432" (UID: "62966127-1cff-4a1a-b9d5-16de42660432"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:36:37.846741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.846724 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62966127-1cff-4a1a-b9d5-16de42660432-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:37.848513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.848482 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62966127-1cff-4a1a-b9d5-16de42660432-kube-api-access-pw6br" pod "62966127-1cff-4a1a-b9d5-16de42660432" (UID: "62966127-1cff-4a1a-b9d5-16de42660432"). InnerVolumeSpecName "kube-api-access-pw6br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:37.947576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:37.947497 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw6br\" (UniqueName: \"kubernetes.io/projected/62966127-1cff-4a1a-b9d5-16de42660432-kube-api-access-pw6br\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:38.642305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:38.642248 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:38.713599 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:38.713533 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt" Nov 04 00:36:38.741123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:38.741078 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt"] Nov 04 00:36:38.747157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:38.747119 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-06ade4-5dcb795cd6-t2mlt"] Nov 04 00:36:39.294664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:39.294624 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62966127-1cff-4a1a-b9d5-16de42660432" path="/var/lib/kubelet/pods/62966127-1cff-4a1a-b9d5-16de42660432/volumes" Nov 04 00:36:39.433955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:39.433913 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:40.359552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:40.359511 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:41.136785 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:41.136750 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:41.923546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:41.923508 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:42.615903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:42.615843 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:42.871907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:42.871817 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:43.279077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:43.279025 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:43.379060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:43.378982 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:43.549854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:43.549763 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:43.652725 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:43.652685 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:43.952221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:43.952124 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:44.276567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:44.276534 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:44.738425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:44.738375 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:45.103270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:45.103178 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:45.607213 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:45.607168 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:45.857452 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:45.857357 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld"] Nov 04 00:36:45.926269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:45.926223 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:46.025693 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:46.025642 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:46.633122 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:46.633084 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:46.744298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:46.744265 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:46.811851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:46.811819 6208 scope.go:122] "RemoveContainer" containerID="0671c050e8723da33a720d98b163d06b30d579e6247146a63cf18a0f209ea583" Nov 04 00:36:46.844581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:46.844549 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:46.944927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:46.944832 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:47.627757 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:47.627712 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:48.607840 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:48.607801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:49.404214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:49.404179 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:49.504365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:49.504332 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:50.239747 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:50.239716 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:50.857842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:50.857802 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" podUID="60d8e725-2cb7-40a0-9acc-bb36b01a0fbd" containerName="payload-processing" containerID="cri-o://722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba" gracePeriod=25 Nov 04 00:36:51.042958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.042922 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:51.142100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.142019 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:51.157473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.157449 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 04 00:36:51.314999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.314971 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-kube-api-access-6d6rt\" (UniqueName: \"kubernetes.io/projected/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-kube-api-access-6d6rt\") pod \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\" (UID: \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\") " Nov 04 00:36:51.315545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.315044 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-plugins-config-volume\") pod \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\" (UID: \"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd\") " Nov 04 00:36:51.315545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.315530 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-plugins-config-volume" pod "60d8e725-2cb7-40a0-9acc-bb36b01a0fbd" (UID: "60d8e725-2cb7-40a0-9acc-bb36b01a0fbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:36:51.317336 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.317309 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-kube-api-access-6d6rt" pod "60d8e725-2cb7-40a0-9acc-bb36b01a0fbd" (UID: "60d8e725-2cb7-40a0-9acc-bb36b01a0fbd"). InnerVolumeSpecName "kube-api-access-6d6rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:51.417012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.416901 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d6rt\" (UniqueName: \"kubernetes.io/projected/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-kube-api-access-6d6rt\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:51.417012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.416944 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:51.787826 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.787786 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" containerID="cri-o://e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512" gracePeriod=45 Nov 04 00:36:51.788069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.788024 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" containerID="cri-o://6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0" gracePeriod=45 Nov 04 00:36:51.792774 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.792734 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": dial tcp 10.133.0.77:8000: connect: connection refused" Nov 04 00:36:51.805672 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.805650 6208 generic.go:356] "Generic (PLEG): container finished" podID="60d8e725-2cb7-40a0-9acc-bb36b01a0fbd" containerID="722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba" exitCode=0 Nov 04 00:36:51.805773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.805706 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" Nov 04 00:36:51.805773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.805720 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" event={"ID":"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd","Type":"ContainerDied","Data":"722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba"} Nov 04 00:36:51.805773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.805761 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld" event={"ID":"60d8e725-2cb7-40a0-9acc-bb36b01a0fbd","Type":"ContainerDied","Data":"e802afa080ca463517ea7aff398377556f451c56d92891efa2b434fedac06418"} Nov 04 00:36:51.805995 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.805784 6208 scope.go:122] "RemoveContainer" containerID="722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba" Nov 04 00:36:51.919821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.919795 6208 scope.go:122] "RemoveContainer" containerID="722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba" Nov 04 00:36:51.920212 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:36:51.920186 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba\": container with ID starting with 722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba not found: ID does not exist" containerID="722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba" Nov 04 00:36:51.920314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.920225 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba"} err="failed to get container status \"722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba\": rpc error: code = NotFound desc = could not find container \"722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba\": container with ID starting with 722aa3d9780aff7c2996dafdc86a4d8ded77dd8f0969525b8757516a42aaf4ba not found: ID does not exist" Nov 04 00:36:51.934382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.934350 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld"] Nov 04 00:36:51.939837 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:51.937809 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c60893-7cfd54746b-ljcld"] Nov 04 00:36:52.080799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.080773 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 04 00:36:52.087168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.087137 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 04 00:36:52.230514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230391 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-tmp-dir\") pod \"8a221959-932c-4943-b9ef-cf46b6e0d895\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " Nov 04 00:36:52.230721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230554 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-kube-api-access-7rlrf\" (UniqueName: \"kubernetes.io/projected/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-kube-api-access-7rlrf\") pod \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " Nov 04 00:36:52.230721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230616 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tls-certs\") pod \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " Nov 04 00:36:52.230721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230694 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-model-cache\") pod \"8a221959-932c-4943-b9ef-cf46b6e0d895\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " Nov 04 00:36:52.231313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230730 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-dshm\") pod \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " Nov 04 00:36:52.231313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230763 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-dshm\") pod \"8a221959-932c-4943-b9ef-cf46b6e0d895\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " Nov 04 00:36:52.231313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230789 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a221959-932c-4943-b9ef-cf46b6e0d895-kube-api-access-snhpt\" (UniqueName: \"kubernetes.io/projected/8a221959-932c-4943-b9ef-cf46b6e0d895-kube-api-access-snhpt\") pod \"8a221959-932c-4943-b9ef-cf46b6e0d895\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " Nov 04 00:36:52.231313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230846 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-model-cache\") pod \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " Nov 04 00:36:52.231313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230942 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a221959-932c-4943-b9ef-cf46b6e0d895-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a221959-932c-4943-b9ef-cf46b6e0d895-tls-certs\") pod \"8a221959-932c-4943-b9ef-cf46b6e0d895\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " Nov 04 00:36:52.231313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.230995 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tmp-dir\") pod \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " Nov 04 00:36:52.231313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.231025 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-home\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-home\") pod \"8a221959-932c-4943-b9ef-cf46b6e0d895\" (UID: \"8a221959-932c-4943-b9ef-cf46b6e0d895\") " Nov 04 00:36:52.231313 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.231080 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-home\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-home\") pod \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\" (UID: \"e6c6643f-325e-491e-a7cf-6ea8357bfe5b\") " Nov 04 00:36:52.234299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.234020 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-tmp-dir" pod "8a221959-932c-4943-b9ef-cf46b6e0d895" (UID: "8a221959-932c-4943-b9ef-cf46b6e0d895"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:52.234459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.234427 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.234634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.234578 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-model-cache" pod "8a221959-932c-4943-b9ef-cf46b6e0d895" (UID: "8a221959-932c-4943-b9ef-cf46b6e0d895"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:52.234942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.234918 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-model-cache" pod "e6c6643f-325e-491e-a7cf-6ea8357bfe5b" (UID: "e6c6643f-325e-491e-a7cf-6ea8357bfe5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:52.235101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.234934 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-home" pod "e6c6643f-325e-491e-a7cf-6ea8357bfe5b" (UID: "e6c6643f-325e-491e-a7cf-6ea8357bfe5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:52.235296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.235269 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tmp-dir" pod "e6c6643f-325e-491e-a7cf-6ea8357bfe5b" (UID: "e6c6643f-325e-491e-a7cf-6ea8357bfe5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:52.235542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.235518 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-home" pod "8a221959-932c-4943-b9ef-cf46b6e0d895" (UID: "8a221959-932c-4943-b9ef-cf46b6e0d895"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:52.236716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.236691 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tls-certs" pod "e6c6643f-325e-491e-a7cf-6ea8357bfe5b" (UID: "e6c6643f-325e-491e-a7cf-6ea8357bfe5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:36:52.237060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.237003 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-dshm" pod "e6c6643f-325e-491e-a7cf-6ea8357bfe5b" (UID: "e6c6643f-325e-491e-a7cf-6ea8357bfe5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:52.237060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.237003 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-kube-api-access-7rlrf" pod "e6c6643f-325e-491e-a7cf-6ea8357bfe5b" (UID: "e6c6643f-325e-491e-a7cf-6ea8357bfe5b"). InnerVolumeSpecName "kube-api-access-7rlrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:52.237207 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.237138 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a221959-932c-4943-b9ef-cf46b6e0d895-kube-api-access-snhpt" pod "8a221959-932c-4943-b9ef-cf46b6e0d895" (UID: "8a221959-932c-4943-b9ef-cf46b6e0d895"). InnerVolumeSpecName "kube-api-access-snhpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:36:52.237207 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.237189 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a221959-932c-4943-b9ef-cf46b6e0d895-tls-certs" pod "8a221959-932c-4943-b9ef-cf46b6e0d895" (UID: "8a221959-932c-4943-b9ef-cf46b6e0d895"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:36:52.237284 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.237270 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-dshm" pod "8a221959-932c-4943-b9ef-cf46b6e0d895" (UID: "8a221959-932c-4943-b9ef-cf46b6e0d895"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:36:52.306173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.306094 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:52.335981 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.335956 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.335981 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.335977 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.336375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.335987 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhpt\" (UniqueName: \"kubernetes.io/projected/8a221959-932c-4943-b9ef-cf46b6e0d895-kube-api-access-snhpt\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.336375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.335998 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.336375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.336009 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a221959-932c-4943-b9ef-cf46b6e0d895-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.336375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.336017 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.336375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.336025 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.336375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.336081 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.336375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.336094 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rlrf\" (UniqueName: \"kubernetes.io/projected/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-kube-api-access-7rlrf\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.336375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.336106 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c6643f-325e-491e-a7cf-6ea8357bfe5b-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.336375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.336114 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a221959-932c-4943-b9ef-cf46b6e0d895-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:36:52.812954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.812920 6208 generic.go:356] "Generic (PLEG): container finished" podID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerID="6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0" exitCode=0 Nov 04 00:36:52.813194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.812966 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" Nov 04 00:36:52.813194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.813009 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" event={"ID":"e6c6643f-325e-491e-a7cf-6ea8357bfe5b","Type":"ContainerDied","Data":"6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0"} Nov 04 00:36:52.813194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.813078 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95" event={"ID":"e6c6643f-325e-491e-a7cf-6ea8357bfe5b","Type":"ContainerDied","Data":"a697b7181e7bda9a32beb59c0b45902a6fbaf814a279cb70ccd2968681ea4820"} Nov 04 00:36:52.813194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.813103 6208 scope.go:122] "RemoveContainer" containerID="6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0" Nov 04 00:36:52.815532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.815507 6208 generic.go:356] "Generic (PLEG): container finished" podID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerID="e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512" exitCode=0 Nov 04 00:36:52.815639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.815544 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" Nov 04 00:36:52.815700 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.815544 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" event={"ID":"8a221959-932c-4943-b9ef-cf46b6e0d895","Type":"ContainerDied","Data":"e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512"} Nov 04 00:36:52.815700 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.815669 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr" event={"ID":"8a221959-932c-4943-b9ef-cf46b6e0d895","Type":"ContainerDied","Data":"50d04c85822bdb511df2a72a435710ef289d9b3ac9882dc227dc85326938b005"} Nov 04 00:36:52.825202 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.825143 6208 scope.go:122] "RemoveContainer" containerID="6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0" Nov 04 00:36:52.825513 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:36:52.825492 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0\": container with ID starting with 6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0 not found: ID does not exist" containerID="6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0" Nov 04 00:36:52.825595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.825520 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0"} err="failed to get container status \"6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0\": rpc error: code = NotFound desc = could not find container \"6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0\": container with ID starting with 6e2818cbc1c588c4ced6268642c77d7a93c16be3929e4789775d518674fc38f0 not found: ID does not exist" Nov 04 00:36:52.825595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.825536 6208 scope.go:122] "RemoveContainer" containerID="e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512" Nov 04 00:36:52.834527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.834510 6208 scope.go:122] "RemoveContainer" containerID="e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512" Nov 04 00:36:52.834794 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:36:52.834775 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512\": container with ID starting with e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512 not found: ID does not exist" containerID="e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512" Nov 04 00:36:52.834849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.834801 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512"} err="failed to get container status \"e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512\": rpc error: code = NotFound desc = could not find container \"e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512\": container with ID starting with e4c38d2265dce4461f194dae725b02b1a13f191590588ed7327f311fc02f9512 not found: ID does not exist" Nov 04 00:36:52.840566 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.840539 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95"] Nov 04 00:36:52.850230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.850209 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/distinct-w3-c60893-kserve-78fd879db7-7tb95"] Nov 04 00:36:52.862174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.862151 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr"] Nov 04 00:36:52.868301 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:52.868279 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c60893/premium-w3-c60893-kserve-6c9c78f87b-9glvr"] Nov 04 00:36:53.036819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:53.036783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:53.137121 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:53.136985 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:53.294099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:53.294063 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d8e725-2cb7-40a0-9acc-bb36b01a0fbd" path="/var/lib/kubelet/pods/60d8e725-2cb7-40a0-9acc-bb36b01a0fbd/volumes" Nov 04 00:36:53.294523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:53.294510 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" path="/var/lib/kubelet/pods/8a221959-932c-4943-b9ef-cf46b6e0d895/volumes" Nov 04 00:36:53.294890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:53.294878 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" path="/var/lib/kubelet/pods/e6c6643f-325e-491e-a7cf-6ea8357bfe5b/volumes" Nov 04 00:36:53.726380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:53.726334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:54.022874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:54.022782 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:54.746275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:54.746235 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:55.294206 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:55.294174 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:55.601110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:55.594638 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:56.538871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:56.538823 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:57.537808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:57.537765 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:57.998575 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:57.998525 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:58.391067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:58.390964 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:58.496080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:58.496026 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:59.234801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:59.234765 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:36:59.948579 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:36:59.948545 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:00.276705 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:00.276668 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:00.375176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:00.375145 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:00.793240 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:00.793207 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:00.905546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:00.905512 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:01.284157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:01.284124 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg"] Nov 04 00:37:01.284636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:01.284340 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" containerID="cri-o://674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b" gracePeriod=30 Nov 04 00:37:01.288389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:01.288362 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:01.592419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:01.592327 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:01.785890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:01.785850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:02.006126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:02.006081 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh"] Nov 04 00:37:02.006364 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:02.006328 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" containerID="cri-o://f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f" gracePeriod=30 Nov 04 00:37:02.052875 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:02.052839 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:02.202456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:02.202412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:02.302765 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:02.302666 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:02.989839 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:02.989799 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:03.090022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:03.089970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:03.938690 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:03.938651 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:04.532076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:04.532021 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:04.723612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:04.723577 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:05.436917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:05.436876 6208 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:37:05.437433 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:05.436959 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:37:05.544998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:05.544956 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:05.705709 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:37:05.705676 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:37:06.328749 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.328718 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:06.543843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.543815 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 04 00:37:06.613510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.613416 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-socket\") pod \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " Nov 04 00:37:06.613510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.613465 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-podinfo\") pod \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " Nov 04 00:37:06.613764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.613528 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-data\") pod \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " Nov 04 00:37:06.613764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.613568 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-certs\") pod \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " Nov 04 00:37:06.613764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.613587 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-envoy\") pod \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " Nov 04 00:37:06.613764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.613619 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-token\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-token\") pod \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " Nov 04 00:37:06.613764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.613659 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istiod-ca-cert\") pod \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " Nov 04 00:37:06.613764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.613748 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-socket" pod "7240cb0b-5340-4a15-afe3-e2ee5f104b85" (UID: "7240cb0b-5340-4a15-afe3-e2ee5f104b85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:06.614091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.613978 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-data" pod "7240cb0b-5340-4a15-afe3-e2ee5f104b85" (UID: "7240cb0b-5340-4a15-afe3-e2ee5f104b85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:06.614159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.614060 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-certs" pod "7240cb0b-5340-4a15-afe3-e2ee5f104b85" (UID: "7240cb0b-5340-4a15-afe3-e2ee5f104b85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:06.614159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.614115 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:06.614248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.614225 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istiod-ca-cert" pod "7240cb0b-5340-4a15-afe3-e2ee5f104b85" (UID: "7240cb0b-5340-4a15-afe3-e2ee5f104b85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:37:06.615971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.615934 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-envoy" pod "7240cb0b-5340-4a15-afe3-e2ee5f104b85" (UID: "7240cb0b-5340-4a15-afe3-e2ee5f104b85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:06.615971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.615944 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-podinfo" pod "7240cb0b-5340-4a15-afe3-e2ee5f104b85" (UID: "7240cb0b-5340-4a15-afe3-e2ee5f104b85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:37:06.615971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.615968 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-token" pod "7240cb0b-5340-4a15-afe3-e2ee5f104b85" (UID: "7240cb0b-5340-4a15-afe3-e2ee5f104b85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:06.715129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.715097 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-kube-api-access-wh2mh\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-kube-api-access-wh2mh\") pod \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " Nov 04 00:37:06.715129 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.715142 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-credential-socket\") pod \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\" (UID: \"7240cb0b-5340-4a15-afe3-e2ee5f104b85\") " Nov 04 00:37:06.715413 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.715402 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:06.715451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.715415 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:06.715451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.715426 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:06.715451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.715438 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:06.715565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.715444 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-credential-socket" pod "7240cb0b-5340-4a15-afe3-e2ee5f104b85" (UID: "7240cb0b-5340-4a15-afe3-e2ee5f104b85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:06.715565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.715451 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:06.715565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.715491 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7240cb0b-5340-4a15-afe3-e2ee5f104b85-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:06.717260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.717238 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-kube-api-access-wh2mh" pod "7240cb0b-5340-4a15-afe3-e2ee5f104b85" (UID: "7240cb0b-5340-4a15-afe3-e2ee5f104b85"). InnerVolumeSpecName "kube-api-access-wh2mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:06.816279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.816233 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2mh\" (UniqueName: \"kubernetes.io/projected/7240cb0b-5340-4a15-afe3-e2ee5f104b85-kube-api-access-wh2mh\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:06.816279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.816272 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7240cb0b-5340-4a15-afe3-e2ee5f104b85-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:06.938628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.938536 6208 generic.go:356] "Generic (PLEG): container finished" podID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerID="674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b" exitCode=0 Nov 04 00:37:06.938628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.938596 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" Nov 04 00:37:06.938829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.938620 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" event={"ID":"7240cb0b-5340-4a15-afe3-e2ee5f104b85","Type":"ContainerDied","Data":"674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b"} Nov 04 00:37:06.938829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.938675 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg" event={"ID":"7240cb0b-5340-4a15-afe3-e2ee5f104b85","Type":"ContainerDied","Data":"cfe8b9e226deac52f8dcef4938f1fbe73260ed65ce69968abb479d0ac8abdc10"} Nov 04 00:37:06.938829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.938699 6208 scope.go:122] "RemoveContainer" containerID="674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b" Nov 04 00:37:06.952889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.952863 6208 scope.go:122] "RemoveContainer" containerID="d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404" Nov 04 00:37:06.962563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.962541 6208 scope.go:122] "RemoveContainer" containerID="674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b" Nov 04 00:37:06.962836 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:37:06.962805 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b\": container with ID starting with 674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b not found: ID does not exist" containerID="674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b" Nov 04 00:37:06.962971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.962845 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b"} err="failed to get container status \"674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b\": rpc error: code = NotFound desc = could not find container \"674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b\": container with ID starting with 674575ae5ac863b6981464f30d6e1e0dcb50be6165b683ac255a146ff085932b not found: ID does not exist" Nov 04 00:37:06.962971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.962867 6208 scope.go:122] "RemoveContainer" containerID="d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404" Nov 04 00:37:06.963184 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:37:06.963162 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404\": container with ID starting with d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404 not found: ID does not exist" containerID="d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404" Nov 04 00:37:06.963244 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.963188 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404"} err="failed to get container status \"d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404\": rpc error: code = NotFound desc = could not find container \"d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404\": container with ID starting with d384a5eb2d9daa3af4381ac4df5148ba52aa0de10cd60646680f880e507f3404 not found: ID does not exist" Nov 04 00:37:06.964425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.964405 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg"] Nov 04 00:37:06.968506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:06.968484 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b3bea7-openshift-default-7984dd78c5-k84fg"] Nov 04 00:37:07.066244 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.066210 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:07.255141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.255116 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 04 00:37:07.294205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.294171 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" path="/var/lib/kubelet/pods/7240cb0b-5340-4a15-afe3-e2ee5f104b85/volumes" Nov 04 00:37:07.319519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.319486 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-certs\") pod \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " Nov 04 00:37:07.319673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.319574 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-kube-api-access-srm9g\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-kube-api-access-srm9g\") pod \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " Nov 04 00:37:07.319673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.319607 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istiod-ca-cert\") pod \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " Nov 04 00:37:07.319802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.319670 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-podinfo\") pod \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " Nov 04 00:37:07.319802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.319741 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-data\") pod \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " Nov 04 00:37:07.319802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.319779 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-token\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-token\") pod \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " Nov 04 00:37:07.319955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.319845 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-envoy\") pod \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " Nov 04 00:37:07.319955 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.319845 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-certs" pod "d6b01bcf-1e5c-4587-8a75-f9734a590e20" (UID: "d6b01bcf-1e5c-4587-8a75-f9734a590e20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:07.320154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.319887 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-socket\") pod \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " Nov 04 00:37:07.320154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320109 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-credential-socket\") pod \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\" (UID: \"d6b01bcf-1e5c-4587-8a75-f9734a590e20\") " Nov 04 00:37:07.320287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320192 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-data" pod "d6b01bcf-1e5c-4587-8a75-f9734a590e20" (UID: "d6b01bcf-1e5c-4587-8a75-f9734a590e20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:07.320287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320191 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istiod-ca-cert" pod "d6b01bcf-1e5c-4587-8a75-f9734a590e20" (UID: "d6b01bcf-1e5c-4587-8a75-f9734a590e20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:37:07.320506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320484 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-credential-socket" pod "d6b01bcf-1e5c-4587-8a75-f9734a590e20" (UID: "d6b01bcf-1e5c-4587-8a75-f9734a590e20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:07.320623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320501 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-socket" pod "d6b01bcf-1e5c-4587-8a75-f9734a590e20" (UID: "d6b01bcf-1e5c-4587-8a75-f9734a590e20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:07.320623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320600 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:07.320623 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320620 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:07.320829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320637 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:07.320829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320652 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:07.320829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.320667 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:07.322463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.322436 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-kube-api-access-srm9g" pod "d6b01bcf-1e5c-4587-8a75-f9734a590e20" (UID: "d6b01bcf-1e5c-4587-8a75-f9734a590e20"). InnerVolumeSpecName "kube-api-access-srm9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:07.322587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.322524 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-envoy" pod "d6b01bcf-1e5c-4587-8a75-f9734a590e20" (UID: "d6b01bcf-1e5c-4587-8a75-f9734a590e20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:07.322587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.322551 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-token" pod "d6b01bcf-1e5c-4587-8a75-f9734a590e20" (UID: "d6b01bcf-1e5c-4587-8a75-f9734a590e20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:07.322587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.322551 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-podinfo" pod "d6b01bcf-1e5c-4587-8a75-f9734a590e20" (UID: "d6b01bcf-1e5c-4587-8a75-f9734a590e20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:37:07.421298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.421266 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srm9g\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-kube-api-access-srm9g\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:07.421298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.421295 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:07.421298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.421306 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:07.421523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.421314 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6b01bcf-1e5c-4587-8a75-f9734a590e20-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:07.838862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.838829 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:07.947464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.947423 6208 generic.go:356] "Generic (PLEG): container finished" podID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerID="f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f" exitCode=0 Nov 04 00:37:07.947661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.947480 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" Nov 04 00:37:07.947661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.947504 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" event={"ID":"d6b01bcf-1e5c-4587-8a75-f9734a590e20","Type":"ContainerDied","Data":"f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f"} Nov 04 00:37:07.947661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.947561 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh" event={"ID":"d6b01bcf-1e5c-4587-8a75-f9734a590e20","Type":"ContainerDied","Data":"adf3bb401c0101808d87617654176b6240b65f5234bff9a7b52e517059973703"} Nov 04 00:37:07.947661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.947585 6208 scope.go:122] "RemoveContainer" containerID="f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f" Nov 04 00:37:07.958434 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.958410 6208 scope.go:122] "RemoveContainer" containerID="f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f" Nov 04 00:37:07.959107 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:37:07.958747 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f\": container with ID starting with f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f not found: ID does not exist" containerID="f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f" Nov 04 00:37:07.959107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.958787 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f"} err="failed to get container status \"f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f\": rpc error: code = NotFound desc = could not find container \"f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f\": container with ID starting with f722d6756495908d86f5df854e53ac5905de3ae981bef3556d60c7005730150f not found: ID does not exist" Nov 04 00:37:07.976205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.976171 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh"] Nov 04 00:37:07.981133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:07.981103 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5738a9-openshift-default-5c8b96d575-xkqdh"] Nov 04 00:37:08.844829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:08.844795 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:08.945195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:08.945162 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:09.294400 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:09.294363 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" path="/var/lib/kubelet/pods/d6b01bcf-1e5c-4587-8a75-f9734a590e20/volumes" Nov 04 00:37:09.652953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:09.652857 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:10.192961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:10.192919 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:10.302557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:10.302521 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:10.658884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:10.658849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:11.394134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:11.394089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:11.497178 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:11.497139 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:11.625487 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:11.625450 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:12.206485 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.206454 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w"] Nov 04 00:37:12.206770 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.206714 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" podUID="e6a19ace-99af-4d67-b658-12655bc033cb" containerName="maas-api" containerID="cri-o://bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d" gracePeriod=30 Nov 04 00:37:12.243486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.243455 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:12.501134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.501106 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 04 00:37:12.586245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.586214 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a19ace-99af-4d67-b658-12655bc033cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a19ace-99af-4d67-b658-12655bc033cb-openshift-service-ca\") pod \"e6a19ace-99af-4d67-b658-12655bc033cb\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " Nov 04 00:37:12.586412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.586308 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-metrics-certs\") pod \"e6a19ace-99af-4d67-b658-12655bc033cb\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " Nov 04 00:37:12.586412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.586343 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-maas-api-tls\") pod \"e6a19ace-99af-4d67-b658-12655bc033cb\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " Nov 04 00:37:12.586412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.586366 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a19ace-99af-4d67-b658-12655bc033cb-kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/e6a19ace-99af-4d67-b658-12655bc033cb-kube-api-access-r5qn7\") pod \"e6a19ace-99af-4d67-b658-12655bc033cb\" (UID: \"e6a19ace-99af-4d67-b658-12655bc033cb\") " Nov 04 00:37:12.586593 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.586567 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a19ace-99af-4d67-b658-12655bc033cb-openshift-service-ca" pod "e6a19ace-99af-4d67-b658-12655bc033cb" (UID: "e6a19ace-99af-4d67-b658-12655bc033cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:37:12.586724 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.586702 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a19ace-99af-4d67-b658-12655bc033cb-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:12.588522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.588490 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a19ace-99af-4d67-b658-12655bc033cb-kube-api-access-r5qn7" pod "e6a19ace-99af-4d67-b658-12655bc033cb" (UID: "e6a19ace-99af-4d67-b658-12655bc033cb"). InnerVolumeSpecName "kube-api-access-r5qn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:12.588522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.588504 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-metrics-certs" pod "e6a19ace-99af-4d67-b658-12655bc033cb" (UID: "e6a19ace-99af-4d67-b658-12655bc033cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:37:12.588647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.588541 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-maas-api-tls" pod "e6a19ace-99af-4d67-b658-12655bc033cb" (UID: "e6a19ace-99af-4d67-b658-12655bc033cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:37:12.644608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.644567 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:12.687878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.687843 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:12.687878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.687874 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a19ace-99af-4d67-b658-12655bc033cb-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:12.688117 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.687886 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qn7\" (UniqueName: \"kubernetes.io/projected/e6a19ace-99af-4d67-b658-12655bc033cb-kube-api-access-r5qn7\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:12.978178 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.978146 6208 generic.go:356] "Generic (PLEG): container finished" podID="e6a19ace-99af-4d67-b658-12655bc033cb" containerID="bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d" exitCode=0 Nov 04 00:37:12.978334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.978198 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" Nov 04 00:37:12.978334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.978228 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" event={"ID":"e6a19ace-99af-4d67-b658-12655bc033cb","Type":"ContainerDied","Data":"bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d"} Nov 04 00:37:12.978334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.978281 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w" event={"ID":"e6a19ace-99af-4d67-b658-12655bc033cb","Type":"ContainerDied","Data":"a5531c865ea33bdcb1fea40856614ee0b707cc5764505fcc0bac60c9855dbd30"} Nov 04 00:37:12.978334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.978303 6208 scope.go:122] "RemoveContainer" containerID="bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d" Nov 04 00:37:12.989105 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.989089 6208 scope.go:122] "RemoveContainer" containerID="bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d" Nov 04 00:37:12.989360 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:37:12.989342 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d\": container with ID starting with bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d not found: ID does not exist" containerID="bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d" Nov 04 00:37:12.989404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:12.989368 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d"} err="failed to get container status \"bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d\": rpc error: code = NotFound desc = could not find container \"bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d\": container with ID starting with bc3ec58d7f97657c24345bc3c5709afa3ae30c5869e75a3c6c1facafba3e943d not found: ID does not exist" Nov 04 00:37:13.002158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:13.002133 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w"] Nov 04 00:37:13.006664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:13.006642 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-15b02a-6d59d96d89-96k5w"] Nov 04 00:37:13.293116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:13.293001 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a19ace-99af-4d67-b658-12655bc033cb" path="/var/lib/kubelet/pods/e6a19ace-99af-4d67-b658-12655bc033cb/volumes" Nov 04 00:37:13.446606 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:13.446573 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:14.152108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:14.152007 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:14.256201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:14.256166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:14.468016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:14.467978 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:15.480586 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:15.480554 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:15.582153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:15.582116 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:16.497522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:16.497479 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:16.602295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:16.602252 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:16.849143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:16.849017 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp"] Nov 04 00:37:16.849342 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:16.849311 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" containerID="cri-o://62d3c92739535121c979fda9adc3fa0cfaf26467014929bb2f927533baa7cad0" gracePeriod=30 Nov 04 00:37:16.883833 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:16.883799 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:17.763513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:17.763482 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:18.301379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:18.301339 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:18.403284 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:18.403250 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:18.712904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:18.712870 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:18.860398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:18.860364 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:19.204141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:19.204051 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl"] Nov 04 00:37:19.204370 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:19.204321 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" containerID="cri-o://98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34" gracePeriod=30 Nov 04 00:37:19.241489 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:19.241456 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:19.740254 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:19.740176 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:19.839595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:19.839555 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:20.664263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:20.664220 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:21.426617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:21.426586 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:22.032350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.032307 6208 generic.go:356] "Generic (PLEG): container finished" podID="e27b7224-8eef-4621-b360-39b48ebee08e" containerID="62d3c92739535121c979fda9adc3fa0cfaf26467014929bb2f927533baa7cad0" exitCode=0 Nov 04 00:37:22.032779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.032415 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" event={"ID":"e27b7224-8eef-4621-b360-39b48ebee08e","Type":"ContainerDied","Data":"62d3c92739535121c979fda9adc3fa0cfaf26467014929bb2f927533baa7cad0"} Nov 04 00:37:22.102560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.102533 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 04 00:37:22.153294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.153211 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:22.191878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.191119 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e27b7224-8eef-4621-b360-39b48ebee08e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7224-8eef-4621-b360-39b48ebee08e-istiod-ca-cert\") pod \"e27b7224-8eef-4621-b360-39b48ebee08e\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " Nov 04 00:37:22.191878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.191195 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-socket\") pod \"e27b7224-8eef-4621-b360-39b48ebee08e\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " Nov 04 00:37:22.191878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.191233 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-credential-socket\") pod \"e27b7224-8eef-4621-b360-39b48ebee08e\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " Nov 04 00:37:22.191878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.191290 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-data\") pod \"e27b7224-8eef-4621-b360-39b48ebee08e\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " Nov 04 00:37:22.191878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.191468 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e27b7224-8eef-4621-b360-39b48ebee08e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7224-8eef-4621-b360-39b48ebee08e-istio-podinfo\") pod \"e27b7224-8eef-4621-b360-39b48ebee08e\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " Nov 04 00:37:22.191878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.191514 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-kube-api-access-9kzrz\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-kube-api-access-9kzrz\") pod \"e27b7224-8eef-4621-b360-39b48ebee08e\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " Nov 04 00:37:22.193392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.192719 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-data" pod "e27b7224-8eef-4621-b360-39b48ebee08e" (UID: "e27b7224-8eef-4621-b360-39b48ebee08e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:22.193392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.192792 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-istio-token\") pod \"e27b7224-8eef-4621-b360-39b48ebee08e\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " Nov 04 00:37:22.193392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.193028 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-envoy\") pod \"e27b7224-8eef-4621-b360-39b48ebee08e\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " Nov 04 00:37:22.193591 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.193393 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-certs\") pod \"e27b7224-8eef-4621-b360-39b48ebee08e\" (UID: \"e27b7224-8eef-4621-b360-39b48ebee08e\") " Nov 04 00:37:22.197260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.194435 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-certs" pod "e27b7224-8eef-4621-b360-39b48ebee08e" (UID: "e27b7224-8eef-4621-b360-39b48ebee08e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:22.197260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.194608 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:22.197260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.194640 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:22.197260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.194871 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-credential-socket" pod "e27b7224-8eef-4621-b360-39b48ebee08e" (UID: "e27b7224-8eef-4621-b360-39b48ebee08e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:22.197260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.197018 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27b7224-8eef-4621-b360-39b48ebee08e-istiod-ca-cert" pod "e27b7224-8eef-4621-b360-39b48ebee08e" (UID: "e27b7224-8eef-4621-b360-39b48ebee08e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:37:22.197984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.197961 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-socket" pod "e27b7224-8eef-4621-b360-39b48ebee08e" (UID: "e27b7224-8eef-4621-b360-39b48ebee08e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:22.200739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.199810 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-kube-api-access-9kzrz" pod "e27b7224-8eef-4621-b360-39b48ebee08e" (UID: "e27b7224-8eef-4621-b360-39b48ebee08e"). InnerVolumeSpecName "kube-api-access-9kzrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:22.202797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.202756 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-envoy" pod "e27b7224-8eef-4621-b360-39b48ebee08e" (UID: "e27b7224-8eef-4621-b360-39b48ebee08e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:22.203120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.203077 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e27b7224-8eef-4621-b360-39b48ebee08e-istio-podinfo" pod "e27b7224-8eef-4621-b360-39b48ebee08e" (UID: "e27b7224-8eef-4621-b360-39b48ebee08e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:37:22.203546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.203488 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-istio-token" pod "e27b7224-8eef-4621-b360-39b48ebee08e" (UID: "e27b7224-8eef-4621-b360-39b48ebee08e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:22.295746 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.295719 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kzrz\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-kube-api-access-9kzrz\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:22.295746 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.295744 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e27b7224-8eef-4621-b360-39b48ebee08e-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:22.295912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.295754 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:22.295912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.295763 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e27b7224-8eef-4621-b360-39b48ebee08e-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:22.295912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.295772 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:22.295912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.295782 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e27b7224-8eef-4621-b360-39b48ebee08e-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:22.295912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.295790 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e27b7224-8eef-4621-b360-39b48ebee08e-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:22.362422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.362386 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:22.462350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.462319 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:22.919139 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:22.919059 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:23.020363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.020318 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:23.039248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.039213 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" event={"ID":"e27b7224-8eef-4621-b360-39b48ebee08e","Type":"ContainerDied","Data":"ed8f9ad3bb93de27cac7932f37046b721fbf3f25f107dc8a848bcb8ec999f155"} Nov 04 00:37:23.039248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.039235 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp" Nov 04 00:37:23.039741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.039283 6208 scope.go:122] "RemoveContainer" containerID="62d3c92739535121c979fda9adc3fa0cfaf26467014929bb2f927533baa7cad0" Nov 04 00:37:23.063493 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.063466 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp"] Nov 04 00:37:23.068243 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.068220 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c60893-openshift-default-f4666cdf-2txdp"] Nov 04 00:37:23.120964 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.120931 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:23.293809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.293770 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" path="/var/lib/kubelet/pods/e27b7224-8eef-4621-b360-39b48ebee08e/volumes" Nov 04 00:37:23.469277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.469243 6208 patch_prober.go:29] interesting pod/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:37:23.469427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.469296 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:37:23.891612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:23.891575 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:24.458397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.458372 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 04 00:37:24.619946 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.619847 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-certs\") pod \"3dfd5078-0143-4c64-8857-e76da9d546de\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " Nov 04 00:37:24.619946 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.619902 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dfd5078-0143-4c64-8857-e76da9d546de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfd5078-0143-4c64-8857-e76da9d546de-istio-podinfo\") pod \"3dfd5078-0143-4c64-8857-e76da9d546de\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " Nov 04 00:37:24.620221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.619957 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-data\") pod \"3dfd5078-0143-4c64-8857-e76da9d546de\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " Nov 04 00:37:24.620221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620071 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-credential-socket\") pod \"3dfd5078-0143-4c64-8857-e76da9d546de\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " Nov 04 00:37:24.620221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620137 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-istio-token\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-istio-token\") pod \"3dfd5078-0143-4c64-8857-e76da9d546de\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " Nov 04 00:37:24.620221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620184 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-socket\") pod \"3dfd5078-0143-4c64-8857-e76da9d546de\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " Nov 04 00:37:24.620448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620233 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-kube-api-access-w6zc7\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-kube-api-access-w6zc7\") pod \"3dfd5078-0143-4c64-8857-e76da9d546de\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " Nov 04 00:37:24.620448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620246 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-certs" pod "3dfd5078-0143-4c64-8857-e76da9d546de" (UID: "3dfd5078-0143-4c64-8857-e76da9d546de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:24.620448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620305 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dfd5078-0143-4c64-8857-e76da9d546de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfd5078-0143-4c64-8857-e76da9d546de-istiod-ca-cert\") pod \"3dfd5078-0143-4c64-8857-e76da9d546de\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " Nov 04 00:37:24.620448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620334 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-data" pod "3dfd5078-0143-4c64-8857-e76da9d546de" (UID: "3dfd5078-0143-4c64-8857-e76da9d546de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:24.620448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620351 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-envoy\") pod \"3dfd5078-0143-4c64-8857-e76da9d546de\" (UID: \"3dfd5078-0143-4c64-8857-e76da9d546de\") " Nov 04 00:37:24.620448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620398 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-credential-socket" pod "3dfd5078-0143-4c64-8857-e76da9d546de" (UID: "3dfd5078-0143-4c64-8857-e76da9d546de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:24.620740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620660 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-socket" pod "3dfd5078-0143-4c64-8857-e76da9d546de" (UID: "3dfd5078-0143-4c64-8857-e76da9d546de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:24.620862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620810 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfd5078-0143-4c64-8857-e76da9d546de-istiod-ca-cert" pod "3dfd5078-0143-4c64-8857-e76da9d546de" (UID: "3dfd5078-0143-4c64-8857-e76da9d546de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:37:24.620862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620831 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:24.620862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620850 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:24.620862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620865 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:24.621137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.620878 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:24.622443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.622417 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-istio-token" pod "3dfd5078-0143-4c64-8857-e76da9d546de" (UID: "3dfd5078-0143-4c64-8857-e76da9d546de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:24.622524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.622479 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-kube-api-access-w6zc7" pod "3dfd5078-0143-4c64-8857-e76da9d546de" (UID: "3dfd5078-0143-4c64-8857-e76da9d546de"). InnerVolumeSpecName "kube-api-access-w6zc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:24.622647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.622628 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dfd5078-0143-4c64-8857-e76da9d546de-istio-podinfo" pod "3dfd5078-0143-4c64-8857-e76da9d546de" (UID: "3dfd5078-0143-4c64-8857-e76da9d546de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:37:24.622704 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.622639 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-envoy" pod "3dfd5078-0143-4c64-8857-e76da9d546de" (UID: "3dfd5078-0143-4c64-8857-e76da9d546de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:37:24.640371 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.640329 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:24.721558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.721523 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:24.721558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.721558 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zc7\" (UniqueName: \"kubernetes.io/projected/3dfd5078-0143-4c64-8857-e76da9d546de-kube-api-access-w6zc7\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:24.721796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.721571 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfd5078-0143-4c64-8857-e76da9d546de-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:24.721796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.721584 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfd5078-0143-4c64-8857-e76da9d546de-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:24.721796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.721599 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfd5078-0143-4c64-8857-e76da9d546de-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:24.838822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:24.838783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:25.053539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.053505 6208 generic.go:356] "Generic (PLEG): container finished" podID="3dfd5078-0143-4c64-8857-e76da9d546de" containerID="98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34" exitCode=0 Nov 04 00:37:25.053733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.053553 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" event={"ID":"3dfd5078-0143-4c64-8857-e76da9d546de","Type":"ContainerDied","Data":"98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34"} Nov 04 00:37:25.053733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.053560 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" Nov 04 00:37:25.053733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.053595 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl" event={"ID":"3dfd5078-0143-4c64-8857-e76da9d546de","Type":"ContainerDied","Data":"685c607bf6985ed33ebc5072d8aea97b5ec1d64d5fe9893c6bd46c672f8689da"} Nov 04 00:37:25.053733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.053617 6208 scope.go:122] "RemoveContainer" containerID="98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34" Nov 04 00:37:25.064469 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.064448 6208 scope.go:122] "RemoveContainer" containerID="c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc" Nov 04 00:37:25.074520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.074494 6208 scope.go:122] "RemoveContainer" containerID="98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34" Nov 04 00:37:25.074793 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:37:25.074770 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34\": container with ID starting with 98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34 not found: ID does not exist" containerID="98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34" Nov 04 00:37:25.074847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.074803 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34"} err="failed to get container status \"98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34\": rpc error: code = NotFound desc = could not find container \"98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34\": container with ID starting with 98dca6e9391281a9b691d7a8749fb9497d3178480d1dc5275e57c87b13b31b34 not found: ID does not exist" Nov 04 00:37:25.074847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.074822 6208 scope.go:122] "RemoveContainer" containerID="c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc" Nov 04 00:37:25.075183 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:37:25.075163 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc\": container with ID starting with c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc not found: ID does not exist" containerID="c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc" Nov 04 00:37:25.075232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.075188 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc"} err="failed to get container status \"c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc\": rpc error: code = NotFound desc = could not find container \"c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc\": container with ID starting with c202bce93461dbf4026e8c4bb98d5623eaf8c0418a514d1051ad4946519f32fc not found: ID does not exist" Nov 04 00:37:25.079433 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.079409 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl"] Nov 04 00:37:25.083153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.083130 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-15b02a-openshift-default-fb8c848bc-t5xrl"] Nov 04 00:37:25.294156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.294120 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" path="/var/lib/kubelet/pods/3dfd5078-0143-4c64-8857-e76da9d546de/volumes" Nov 04 00:37:25.353648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.353569 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:25.660024 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:25.659901 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:26.460663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:26.460624 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:27.249752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:27.249695 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:28.219246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:28.219207 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:29.041626 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:29.041590 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:29.731127 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:29.731090 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:30.531359 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:30.531320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:31.291231 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:31.291192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:31.921105 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:31.921062 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6"] Nov 04 00:37:31.931383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:31.931348 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs"] Nov 04 00:37:31.993944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:31.993884 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:32.093688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:32.093645 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:32.925916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:32.925884 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:33.100547 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:33.100507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:35.705805 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:37:35.705765 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:37:36.922374 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:36.922317 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" podUID="1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb" containerName="payload-processing" containerID="cri-o://7e6cd475fab8c6dd82cf71e37b1dc66468d8e5b7795bdca4deed4033685d49dc" gracePeriod=25 Nov 04 00:37:36.931970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:36.931909 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" podUID="b6e29d36-f359-48ed-bebb-15951255c782" containerName="payload-pre-processing" containerID="cri-o://ac3e94022d1240a491f96ab3d9b06fd91765b09e55f820bce2513f63235f316f" gracePeriod=25 Nov 04 00:37:37.131960 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.131777 6208 generic.go:356] "Generic (PLEG): container finished" podID="b6e29d36-f359-48ed-bebb-15951255c782" containerID="ac3e94022d1240a491f96ab3d9b06fd91765b09e55f820bce2513f63235f316f" exitCode=0 Nov 04 00:37:37.131960 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.131888 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" event={"ID":"b6e29d36-f359-48ed-bebb-15951255c782","Type":"ContainerDied","Data":"ac3e94022d1240a491f96ab3d9b06fd91765b09e55f820bce2513f63235f316f"} Nov 04 00:37:37.133927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.133898 6208 generic.go:356] "Generic (PLEG): container finished" podID="1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb" containerID="7e6cd475fab8c6dd82cf71e37b1dc66468d8e5b7795bdca4deed4033685d49dc" exitCode=0 Nov 04 00:37:37.134087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.133975 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" event={"ID":"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb","Type":"ContainerDied","Data":"7e6cd475fab8c6dd82cf71e37b1dc66468d8e5b7795bdca4deed4033685d49dc"} Nov 04 00:37:37.195382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.195358 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 04 00:37:37.234953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.234928 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 04 00:37:37.261634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.261607 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-kube-api-access-g8srs\" (UniqueName: \"kubernetes.io/projected/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-kube-api-access-g8srs\") pod \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\" (UID: \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\") " Nov 04 00:37:37.261761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.261660 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-plugins-config-volume\") pod \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\" (UID: \"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb\") " Nov 04 00:37:37.261847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.261828 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e29d36-f359-48ed-bebb-15951255c782-kube-api-access-fxkj4\" (UniqueName: \"kubernetes.io/projected/b6e29d36-f359-48ed-bebb-15951255c782-kube-api-access-fxkj4\") pod \"b6e29d36-f359-48ed-bebb-15951255c782\" (UID: \"b6e29d36-f359-48ed-bebb-15951255c782\") " Nov 04 00:37:37.261976 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.261917 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e29d36-f359-48ed-bebb-15951255c782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e29d36-f359-48ed-bebb-15951255c782-plugins-config-volume\") pod \"b6e29d36-f359-48ed-bebb-15951255c782\" (UID: \"b6e29d36-f359-48ed-bebb-15951255c782\") " Nov 04 00:37:37.262105 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.262071 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-plugins-config-volume" pod "1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb" (UID: "1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:37:37.262399 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.262376 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:37.262502 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.262376 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e29d36-f359-48ed-bebb-15951255c782-plugins-config-volume" pod "b6e29d36-f359-48ed-bebb-15951255c782" (UID: "b6e29d36-f359-48ed-bebb-15951255c782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:37:37.263819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.263794 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-kube-api-access-g8srs" pod "1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb" (UID: "1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb"). InnerVolumeSpecName "kube-api-access-g8srs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:37.263895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.263846 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e29d36-f359-48ed-bebb-15951255c782-kube-api-access-fxkj4" pod "b6e29d36-f359-48ed-bebb-15951255c782" (UID: "b6e29d36-f359-48ed-bebb-15951255c782"). InnerVolumeSpecName "kube-api-access-fxkj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:37.362950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.362915 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8srs\" (UniqueName: \"kubernetes.io/projected/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb-kube-api-access-g8srs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:37.362950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.362944 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxkj4\" (UniqueName: \"kubernetes.io/projected/b6e29d36-f359-48ed-bebb-15951255c782-kube-api-access-fxkj4\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:37.363142 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:37.362958 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6e29d36-f359-48ed-bebb-15951255c782-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:38.023559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.023521 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:38.140314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.140279 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" Nov 04 00:37:38.140314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.140298 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs" event={"ID":"b6e29d36-f359-48ed-bebb-15951255c782","Type":"ContainerDied","Data":"d08c7a5fb2316fe533464cb6345fcc299793f4027130fa2e99e21a0bf96fb36f"} Nov 04 00:37:38.140530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.140360 6208 scope.go:122] "RemoveContainer" containerID="ac3e94022d1240a491f96ab3d9b06fd91765b09e55f820bce2513f63235f316f" Nov 04 00:37:38.141673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.141656 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" Nov 04 00:37:38.141771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.141694 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6" event={"ID":"1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb","Type":"ContainerDied","Data":"8998e4382086b761cd7104be19113b1385b69d2bfeff41d4a61edb09a91e9771"} Nov 04 00:37:38.150140 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.150123 6208 scope.go:122] "RemoveContainer" containerID="7e6cd475fab8c6dd82cf71e37b1dc66468d8e5b7795bdca4deed4033685d49dc" Nov 04 00:37:38.159812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.159787 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs"] Nov 04 00:37:38.165180 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.165156 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-80f8ba-6bc6d848c-ncqcs"] Nov 04 00:37:38.177224 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.177205 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6"] Nov 04 00:37:38.182177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:38.182157 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-80f8ba-85c755b788-5f4w6"] Nov 04 00:37:39.154308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:39.154256 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:39.292809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:39.292772 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb" path="/var/lib/kubelet/pods/1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb/volumes" Nov 04 00:37:39.293215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:39.293198 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e29d36-f359-48ed-bebb-15951255c782" path="/var/lib/kubelet/pods/b6e29d36-f359-48ed-bebb-15951255c782/volumes" Nov 04 00:37:43.531370 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.531332 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg"] Nov 04 00:37:43.532065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.531541 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" podUID="ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" containerName="maas-api" containerID="cri-o://844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba" gracePeriod=30 Nov 04 00:37:43.581488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.581442 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:43.810897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.810870 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 04 00:37:43.934000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.933964 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-kube-api-access-cn6fc\" (UniqueName: \"kubernetes.io/projected/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-kube-api-access-cn6fc\") pod \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " Nov 04 00:37:43.934212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.934091 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-openshift-service-ca\") pod \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " Nov 04 00:37:43.934212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.934160 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-maas-api-tls\") pod \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " Nov 04 00:37:43.934341 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.934210 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-metrics-certs\") pod \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\" (UID: \"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5\") " Nov 04 00:37:43.934521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.934494 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-openshift-service-ca" pod "ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" (UID: "ca0f1c2d-87d1-4478-9e3a-4eed792f66d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:37:43.936233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.936206 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-kube-api-access-cn6fc" pod "ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" (UID: "ca0f1c2d-87d1-4478-9e3a-4eed792f66d5"). InnerVolumeSpecName "kube-api-access-cn6fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:37:43.936326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.936278 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-maas-api-tls" pod "ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" (UID: "ca0f1c2d-87d1-4478-9e3a-4eed792f66d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:37:43.936326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:43.936300 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-metrics-certs" pod "ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" (UID: "ca0f1c2d-87d1-4478-9e3a-4eed792f66d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:37:44.035379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.035348 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:44.035379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.035376 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:44.035549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.035387 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:44.035549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.035397 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn6fc\" (UniqueName: \"kubernetes.io/projected/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5-kube-api-access-cn6fc\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:37:44.181264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.181185 6208 generic.go:356] "Generic (PLEG): container finished" podID="ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" containerID="844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba" exitCode=0 Nov 04 00:37:44.181264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.181238 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" Nov 04 00:37:44.181457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.181265 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" event={"ID":"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5","Type":"ContainerDied","Data":"844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba"} Nov 04 00:37:44.181457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.181307 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg" event={"ID":"ca0f1c2d-87d1-4478-9e3a-4eed792f66d5","Type":"ContainerDied","Data":"bd3e56e3b3e48bdeb5d4cdb91ba7953848c11d85a8b0eb908f74618a26035762"} Nov 04 00:37:44.181457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.181330 6208 scope.go:122] "RemoveContainer" containerID="844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba" Nov 04 00:37:44.192605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.192585 6208 scope.go:122] "RemoveContainer" containerID="844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba" Nov 04 00:37:44.192886 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:37:44.192863 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba\": container with ID starting with 844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba not found: ID does not exist" containerID="844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba" Nov 04 00:37:44.192941 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.192895 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba"} err="failed to get container status \"844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba\": rpc error: code = NotFound desc = could not find container \"844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba\": container with ID starting with 844d7aa7b7bcbdbd1f609be947632d79ba7e033d22ff4eee251d8e1b00cc70ba not found: ID does not exist" Nov 04 00:37:44.204406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.204383 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg"] Nov 04 00:37:44.214347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:44.210082 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-80f8ba-67f65c687-5zngg"] Nov 04 00:37:45.293639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:45.293601 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" path="/var/lib/kubelet/pods/ca0f1c2d-87d1-4478-9e3a-4eed792f66d5/volumes" Nov 04 00:37:45.328627 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:45.328588 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:45.528665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:45.528627 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:52.202269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:52.202229 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:37:58.054540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:37:58.054501 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:02.439318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:02.439281 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:02.798018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:02.797958 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:02.805912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:02.805881 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j"] Nov 04 00:38:02.806159 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:02.806131 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" containerID="cri-o://59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb" gracePeriod=30 Nov 04 00:38:02.896096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:02.896027 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:03.686591 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:03.686562 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:03.999674 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:03.999643 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b594799b-mkv2k_93e48259-862a-4633-b927-84fd5351d25a/authorino/0.log" Nov 04 00:38:04.585463 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:04.585408 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:04.656417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:04.656389 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jc64k_fd69111b-a7ff-4661-8608-4dbc46282b48/kuadrant-console-plugin/0.log" Nov 04 00:38:05.704290 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:38:05.704255 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:38:06.071293 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:06.071225 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:06.171018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:06.170978 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:08.054200 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.054174 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 04 00:38:08.077242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077189 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-token\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-token\") pod \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " Nov 04 00:38:08.077242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077231 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-credential-socket\") pod \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " Nov 04 00:38:08.077407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077298 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-kube-api-access-k4f5p\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-kube-api-access-k4f5p\") pod \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " Nov 04 00:38:08.077407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077320 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-data\") pod \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " Nov 04 00:38:08.077407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077340 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-certs\") pod \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " Nov 04 00:38:08.077571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077408 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istiod-ca-cert\") pod \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " Nov 04 00:38:08.077571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077475 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-socket\") pod \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " Nov 04 00:38:08.077571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077517 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-envoy\") pod \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " Nov 04 00:38:08.077571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077544 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-podinfo\") pod \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\" (UID: \"f6e6bc29-8cd4-4910-9a12-3686b55c19b4\") " Nov 04 00:38:08.077786 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077698 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-data" pod "f6e6bc29-8cd4-4910-9a12-3686b55c19b4" (UID: "f6e6bc29-8cd4-4910-9a12-3686b55c19b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:38:08.077786 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077715 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-credential-socket" pod "f6e6bc29-8cd4-4910-9a12-3686b55c19b4" (UID: "f6e6bc29-8cd4-4910-9a12-3686b55c19b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:38:08.077786 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077742 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-certs" pod "f6e6bc29-8cd4-4910-9a12-3686b55c19b4" (UID: "f6e6bc29-8cd4-4910-9a12-3686b55c19b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:38:08.077965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077791 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-socket" pod "f6e6bc29-8cd4-4910-9a12-3686b55c19b4" (UID: "f6e6bc29-8cd4-4910-9a12-3686b55c19b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:38:08.077965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077887 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:08.077965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077921 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:08.077965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077939 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:08.077965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.077952 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:08.078335 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.078297 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istiod-ca-cert" pod "f6e6bc29-8cd4-4910-9a12-3686b55c19b4" (UID: "f6e6bc29-8cd4-4910-9a12-3686b55c19b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:38:08.080009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.079970 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-token" pod "f6e6bc29-8cd4-4910-9a12-3686b55c19b4" (UID: "f6e6bc29-8cd4-4910-9a12-3686b55c19b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:38:08.080276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.080253 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-kube-api-access-k4f5p" pod "f6e6bc29-8cd4-4910-9a12-3686b55c19b4" (UID: "f6e6bc29-8cd4-4910-9a12-3686b55c19b4"). InnerVolumeSpecName "kube-api-access-k4f5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:38:08.080670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.080628 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-podinfo" pod "f6e6bc29-8cd4-4910-9a12-3686b55c19b4" (UID: "f6e6bc29-8cd4-4910-9a12-3686b55c19b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:38:08.081210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.081177 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-envoy" pod "f6e6bc29-8cd4-4910-9a12-3686b55c19b4" (UID: "f6e6bc29-8cd4-4910-9a12-3686b55c19b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:38:08.178556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.178525 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4f5p\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-kube-api-access-k4f5p\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:08.178556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.178554 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:08.178739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.178563 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:08.178739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.178574 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:08.178739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.178583 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e6bc29-8cd4-4910-9a12-3686b55c19b4-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:08.311111 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.311081 6208 generic.go:356] "Generic (PLEG): container finished" podID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerID="59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb" exitCode=0 Nov 04 00:38:08.311310 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.311126 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" Nov 04 00:38:08.311310 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.311168 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" event={"ID":"f6e6bc29-8cd4-4910-9a12-3686b55c19b4","Type":"ContainerDied","Data":"59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb"} Nov 04 00:38:08.311310 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.311225 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j" event={"ID":"f6e6bc29-8cd4-4910-9a12-3686b55c19b4","Type":"ContainerDied","Data":"455ad7903d129880cafcdda41d7d609ceb253db2e03a1034b18866ded51dee28"} Nov 04 00:38:08.311310 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.311256 6208 scope.go:122] "RemoveContainer" containerID="59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb" Nov 04 00:38:08.321768 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.321750 6208 scope.go:122] "RemoveContainer" containerID="59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb" Nov 04 00:38:08.322050 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:38:08.322020 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb\": container with ID starting with 59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb not found: ID does not exist" containerID="59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb" Nov 04 00:38:08.322124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.322060 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb"} err="failed to get container status \"59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb\": rpc error: code = NotFound desc = could not find container \"59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb\": container with ID starting with 59707699475c83436640423a65e079ee5e13156c98599424ae825eee9df33deb not found: ID does not exist" Nov 04 00:38:08.334436 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.334386 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j"] Nov 04 00:38:08.340405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.340376 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-80f8ba-openshift-default-6ffc9759f5-2648j"] Nov 04 00:38:08.956023 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:08.955975 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:09.294426 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:09.294350 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" path="/var/lib/kubelet/pods/f6e6bc29-8cd4-4910-9a12-3686b55c19b4/volumes" Nov 04 00:38:09.596905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:09.596794 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:09.696796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:09.696755 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:16.949429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:16.949390 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/1.log" Nov 04 00:38:29.118631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:29.118598 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/0.log" Nov 04 00:38:34.193895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:34.193855 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:35.703257 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:38:35.703222 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:38:36.380887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:36.380849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:37.405557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:37.405521 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:37.507086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:37.507027 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:38.606150 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:38.606112 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:38.715438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:38.715405 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:38.819503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:38.818707 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:39.532636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:39.532599 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:40.325885 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:40.325850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:41.149668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:41.149633 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:41.922473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:41.922420 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:42.637231 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:42.637190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:43.351438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:43.351401 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:44.151276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:44.151246 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:44.929126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:44.929089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:45.639654 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:45.639621 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:46.433903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:46.433858 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:46.584669 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:46.584629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:47.407285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:47.407242 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:48.329787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:48.329751 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:49.052426 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:49.052385 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:49.829185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:49.829107 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:50.577837 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:50.577802 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:51.456441 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:51.456408 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:52.234409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:52.234373 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:53.049821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.049783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:53.140457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140416 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n"] Nov 04 00:38:53.140875 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140856 6208 cpu_manager.go:414] "containerMap: removing container" podUID="127f6328-65eb-4bea-898b-f01e05d6a6dc" containerName="payload-processing" Nov 04 00:38:53.140875 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140875 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127f6328-65eb-4bea-898b-f01e05d6a6dc" containerName="payload-processing" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140887 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140893 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140902 6208 cpu_manager.go:414] "containerMap: removing container" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140907 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140914 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140919 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140929 6208 cpu_manager.go:414] "containerMap: removing container" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140935 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140941 6208 cpu_manager.go:414] "containerMap: removing container" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140947 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140957 6208 cpu_manager.go:414] "containerMap: removing container" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140964 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140970 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerName="cleanup" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140975 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerName="cleanup" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140982 6208 cpu_manager.go:414] "containerMap: removing container" podUID="204d311a-3628-48a3-98f1-36cf66bc12bc" containerName="maas-api" Nov 04 00:38:53.140979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140986 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204d311a-3628-48a3-98f1-36cf66bc12bc" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140993 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.140997 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141004 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141010 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141020 6208 cpu_manager.go:414] "containerMap: removing container" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141025 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141044 6208 cpu_manager.go:414] "containerMap: removing container" podUID="d2fe6082-38e3-4e74-8712-4a1e9977cfaa" containerName="payload-pre-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141051 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fe6082-38e3-4e74-8712-4a1e9977cfaa" containerName="payload-pre-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141062 6208 cpu_manager.go:414] "containerMap: removing container" podUID="49e453cb-0248-4842-936e-c1e2716d822c" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141068 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e453cb-0248-4842-936e-c1e2716d822c" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141082 6208 cpu_manager.go:414] "containerMap: removing container" podUID="60d8e725-2cb7-40a0-9acc-bb36b01a0fbd" containerName="payload-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141090 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d8e725-2cb7-40a0-9acc-bb36b01a0fbd" containerName="payload-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141101 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141108 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141116 6208 cpu_manager.go:414] "containerMap: removing container" podUID="79986d29-e237-4bbc-bb59-9687d182ca4c" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141121 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79986d29-e237-4bbc-bb59-9687d182ca4c" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141128 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141132 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141138 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141142 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141148 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5a7f005b-f784-4e32-9eee-6582056d16fe" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141152 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7f005b-f784-4e32-9eee-6582056d16fe" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141160 6208 cpu_manager.go:414] "containerMap: removing container" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141165 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141173 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5b675c79-e14d-440f-b32b-24dd6bd05427" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141178 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b675c79-e14d-440f-b32b-24dd6bd05427" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141185 6208 cpu_manager.go:414] "containerMap: removing container" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141190 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141198 6208 cpu_manager.go:414] "containerMap: removing container" podUID="da7d10bd-27ea-429e-8730-1b67bb6e7e46" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141202 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7d10bd-27ea-429e-8730-1b67bb6e7e46" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141210 6208 cpu_manager.go:414] "containerMap: removing container" podUID="49e453cb-0248-4842-936e-c1e2716d822c" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141214 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e453cb-0248-4842-936e-c1e2716d822c" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141222 6208 cpu_manager.go:414] "containerMap: removing container" podUID="ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141227 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141233 6208 cpu_manager.go:414] "containerMap: removing container" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141237 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141244 6208 cpu_manager.go:414] "containerMap: removing container" podUID="6fa02593-a635-47a7-a21f-9d67b1e8018e" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141249 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa02593-a635-47a7-a21f-9d67b1e8018e" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141254 6208 cpu_manager.go:414] "containerMap: removing container" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141258 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141264 6208 cpu_manager.go:414] "containerMap: removing container" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141268 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141276 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141281 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141289 6208 cpu_manager.go:414] "containerMap: removing container" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141293 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141299 6208 cpu_manager.go:414] "containerMap: removing container" podUID="49e453cb-0248-4842-936e-c1e2716d822c" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141303 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e453cb-0248-4842-936e-c1e2716d822c" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141310 6208 cpu_manager.go:414] "containerMap: removing container" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141315 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141320 6208 cpu_manager.go:414] "containerMap: removing container" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141325 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141333 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1824481f-5659-4932-ab0b-6726f418b3a0" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141337 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1824481f-5659-4932-ab0b-6726f418b3a0" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141345 6208 cpu_manager.go:414] "containerMap: removing container" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141350 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141357 6208 cpu_manager.go:414] "containerMap: removing container" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141363 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141369 6208 cpu_manager.go:414] "containerMap: removing container" podUID="64445549-ad95-4060-9522-ab6b550cecbf" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141374 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64445549-ad95-4060-9522-ab6b550cecbf" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141379 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141384 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141389 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e53bfbe1-15dc-4b22-8121-5e927de1be27" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141393 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53bfbe1-15dc-4b22-8121-5e927de1be27" containerName="maas-api" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141400 6208 cpu_manager.go:414] "containerMap: removing container" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141404 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141410 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb" containerName="payload-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141414 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb" containerName="payload-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141423 6208 cpu_manager.go:414] "containerMap: removing container" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141428 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141434 6208 cpu_manager.go:414] "containerMap: removing container" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141439 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141445 6208 cpu_manager.go:414] "containerMap: removing container" podUID="b6e29d36-f359-48ed-bebb-15951255c782" containerName="payload-pre-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141449 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e29d36-f359-48ed-bebb-15951255c782" containerName="payload-pre-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141454 6208 cpu_manager.go:414] "containerMap: removing container" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141459 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141469 6208 cpu_manager.go:414] "containerMap: removing container" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141473 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141481 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141486 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141492 6208 cpu_manager.go:414] "containerMap: removing container" podUID="62966127-1cff-4a1a-b9d5-16de42660432" containerName="payload-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141496 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62966127-1cff-4a1a-b9d5-16de42660432" containerName="payload-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141502 6208 cpu_manager.go:414] "containerMap: removing container" podUID="cabc3bde-6478-4a87-af40-513329416597" containerName="payload-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141506 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabc3bde-6478-4a87-af40-513329416597" containerName="payload-processing" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141513 6208 cpu_manager.go:414] "containerMap: removing container" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141517 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141523 6208 cpu_manager.go:414] "containerMap: removing container" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141528 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141535 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141540 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141546 6208 cpu_manager.go:414] "containerMap: removing container" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerName="cleanup" Nov 04 00:38:53.141506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141551 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141561 6208 cpu_manager.go:414] "containerMap: removing container" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141566 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141571 6208 cpu_manager.go:414] "containerMap: removing container" podUID="b62d2e16-32d3-46b2-8795-4fed31a1c458" containerName="payload-pre-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141575 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62d2e16-32d3-46b2-8795-4fed31a1c458" containerName="payload-pre-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141582 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e6a19ace-99af-4d67-b658-12655bc033cb" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141587 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a19ace-99af-4d67-b658-12655bc033cb" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141592 6208 cpu_manager.go:414] "containerMap: removing container" podUID="2d4acee3-e8e1-4257-90bd-8aab3f96e757" containerName="payload-pre-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141597 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4acee3-e8e1-4257-90bd-8aab3f96e757" containerName="payload-pre-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141604 6208 cpu_manager.go:414] "containerMap: removing container" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141608 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141616 6208 cpu_manager.go:414] "containerMap: removing container" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141620 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141689 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141697 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141704 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141710 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="62966127-1cff-4a1a-b9d5-16de42660432" containerName="payload-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141717 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabc3bde-6478-4a87-af40-513329416597" containerName="payload-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141722 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141728 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141733 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a221959-932c-4943-b9ef-cf46b6e0d895" containerName="main" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141738 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c6643f-325e-491e-a7cf-6ea8357bfe5b" containerName="main" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141742 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141747 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141753 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b01bcf-1e5c-4587-8a75-f9734a590e20" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141758 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62d2e16-32d3-46b2-8795-4fed31a1c458" containerName="payload-pre-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141763 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a19ace-99af-4d67-b658-12655bc033cb" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141769 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4acee3-e8e1-4257-90bd-8aab3f96e757" containerName="payload-pre-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141774 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141778 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141784 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141790 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="127f6328-65eb-4bea-898b-f01e05d6a6dc" containerName="payload-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141796 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a66e1c08-6769-477b-bb4a-1970e09e6627" containerName="main" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141802 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="88cecb6a-ebee-4eb2-abac-1cc23957ae3d" containerName="main" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141808 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141812 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141817 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9fe7e3-20d0-4b79-b147-ba9f6fb319fb" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141823 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="204d311a-3628-48a3-98f1-36cf66bc12bc" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141828 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141833 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31de50e-34be-4e03-8c70-e98d3d4cb577" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141838 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141843 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fe6082-38e3-4e74-8712-4a1e9977cfaa" containerName="payload-pre-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141847 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e453cb-0248-4842-936e-c1e2716d822c" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141852 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d8e725-2cb7-40a0-9acc-bb36b01a0fbd" containerName="payload-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141858 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e6bc29-8cd4-4910-9a12-3686b55c19b4" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141864 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="79986d29-e237-4bbc-bb59-9687d182ca4c" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141869 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27b7224-8eef-4621-b360-39b48ebee08e" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141873 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141879 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7f005b-f784-4e32-9eee-6582056d16fe" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141883 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141888 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b675c79-e14d-440f-b32b-24dd6bd05427" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141893 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29233c7-5e58-4ed7-b12c-6aa0ffe49e79" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141900 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7d10bd-27ea-429e-8730-1b67bb6e7e46" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141904 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e453cb-0248-4842-936e-c1e2716d822c" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141909 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141913 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0f1c2d-87d1-4478-9e3a-4eed792f66d5" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141918 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141923 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa02593-a635-47a7-a21f-9d67b1e8018e" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141929 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141934 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c01b66-326c-41c4-b612-5551e8af6e85" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141939 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141946 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="7240cb0b-5340-4a15-afe3-e2ee5f104b85" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141951 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141956 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e453cb-0248-4842-936e-c1e2716d822c" containerName="cleanup" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141960 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b9470d-1121-4176-be18-bd5348ae17db" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141966 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141972 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1824481f-5659-4932-ab0b-6726f418b3a0" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141977 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d2d360-21ba-4c73-b6a1-fe28276a6f2f" containerName="main" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141983 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="64445549-ad95-4060-9522-ab6b550cecbf" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141988 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e200da9d-3c7d-4904-9d6d-88badac0a185" containerName="main" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141992 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.141998 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53bfbe1-15dc-4b22-8121-5e927de1be27" containerName="maas-api" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.142003 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3cc16a-8f73-49a2-8aec-87ebbb3cc1eb" containerName="payload-processing" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.142008 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1457a0f-48fe-45d4-a180-ae1d84f549ba" containerName="istio-proxy" Nov 04 00:38:53.144541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.142014 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e29d36-f359-48ed-bebb-15951255c782" containerName="payload-pre-processing" Nov 04 00:38:53.146985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.146081 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.152519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.152493 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:38:53.155392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.155366 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n"] Nov 04 00:38:53.155973 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.155953 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:38:53.156123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.156016 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:38:53.156196 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.156142 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g8gzq\"" Nov 04 00:38:53.223849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.223815 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ck5x\" (UniqueName: \"kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.223849 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.223851 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.224102 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.223881 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.324942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.324860 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ck5x\" (UniqueName: \"kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.324942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.324904 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.325213 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.324946 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.325646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.325620 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.327297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.327270 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.332489 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:38:53.332469 6208 projected.go:196] Error preparing data for projected volume kube-api-access-9ck5x for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 04 00:38:53.333058 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:38:53.333024 6208 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x podName:172c5038-00d7-4378-b056-3fe07ec00b16 nodeName:}" failed. No retries permitted until 2026-11-04 00:38:53.833004707 +0000 UTC m=+4152.655600862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9ck5x" (UniqueName: "kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x") pod "maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" (UID: "172c5038-00d7-4378-b056-3fe07ec00b16") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 04 00:38:53.856952 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.856919 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:53.929094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.929057 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ck5x\" (UniqueName: \"kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:53.941889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:53.941857 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ck5x\" (UniqueName: \"kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x\") pod \"maas-api-revoke-keys-e2e-ait-849709d8-l4c7n\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:54.063138 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:54.063102 6208 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" secret="" err="secret \"maas-api-cleanup-dockercfg-g8gzq\" not found" Nov 04 00:38:54.063546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:54.063170 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:54.341678 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:54.341591 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n"] Nov 04 00:38:54.344080 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:38:54.344019 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172c5038_00d7_4378_b056_3fe07ec00b16.slice/crio-d47ef73a9174f61405922276bfc88b2464aa8b0b12abc0c0a0046590449f02ae WatchSource:0}: Error finding container d47ef73a9174f61405922276bfc88b2464aa8b0b12abc0c0a0046590449f02ae: Status 404 returned error can't find the container with id d47ef73a9174f61405922276bfc88b2464aa8b0b12abc0c0a0046590449f02ae Nov 04 00:38:54.546887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:54.546845 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" event={"ID":"172c5038-00d7-4378-b056-3fe07ec00b16","Type":"ContainerStarted","Data":"d47ef73a9174f61405922276bfc88b2464aa8b0b12abc0c0a0046590449f02ae"} Nov 04 00:38:54.738530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:54.738474 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:55.459895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:55.459846 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:55.553546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:55.553507 6208 generic.go:356] "Generic (PLEG): container finished" podID="172c5038-00d7-4378-b056-3fe07ec00b16" containerID="f2ab63468845e8fc23cd2ab95a6cd4a78fbe9bbe66ead1436724a4ac4647ce20" exitCode=0 Nov 04 00:38:55.553724 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:55.553589 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" event={"ID":"172c5038-00d7-4378-b056-3fe07ec00b16","Type":"ContainerDied","Data":"f2ab63468845e8fc23cd2ab95a6cd4a78fbe9bbe66ead1436724a4ac4647ce20"} Nov 04 00:38:56.324446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.324404 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:56.711156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.711131 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:56.758194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.758162 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x\" (UniqueName: \"kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x\") pod \"172c5038-00d7-4378-b056-3fe07ec00b16\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " Nov 04 00:38:56.758362 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.758238 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-openshift-ca\") pod \"172c5038-00d7-4378-b056-3fe07ec00b16\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " Nov 04 00:38:56.758362 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.758279 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-xks-ca\") pod \"172c5038-00d7-4378-b056-3fe07ec00b16\" (UID: \"172c5038-00d7-4378-b056-3fe07ec00b16\") " Nov 04 00:38:56.758658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.758625 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-openshift-ca" pod "172c5038-00d7-4378-b056-3fe07ec00b16" (UID: "172c5038-00d7-4378-b056-3fe07ec00b16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:38:56.760376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.760347 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-xks-ca" pod "172c5038-00d7-4378-b056-3fe07ec00b16" (UID: "172c5038-00d7-4378-b056-3fe07ec00b16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:38:56.760490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.760355 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x" pod "172c5038-00d7-4378-b056-3fe07ec00b16" (UID: "172c5038-00d7-4378-b056-3fe07ec00b16"). InnerVolumeSpecName "kube-api-access-9ck5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:38:56.860124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.860024 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ck5x\" (UniqueName: \"kubernetes.io/projected/172c5038-00d7-4378-b056-3fe07ec00b16-kube-api-access-9ck5x\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:56.860124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.860075 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:56.860124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:56.860086 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172c5038-00d7-4378-b056-3fe07ec00b16-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:38:57.057094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:57.057056 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:57.565226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:57.565145 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" event={"ID":"172c5038-00d7-4378-b056-3fe07ec00b16","Type":"ContainerDied","Data":"d47ef73a9174f61405922276bfc88b2464aa8b0b12abc0c0a0046590449f02ae"} Nov 04 00:38:57.565226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:57.565173 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n" Nov 04 00:38:57.565226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:57.565193 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d47ef73a9174f61405922276bfc88b2464aa8b0b12abc0c0a0046590449f02ae" Nov 04 00:38:57.840326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:57.840235 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:58.113420 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:58.113324 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:58.630910 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:58.630869 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:38:59.351847 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:38:59.351808 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:00.132678 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:00.132632 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:00.922664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:00.922628 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:01.745966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:01.745922 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:02.537805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:02.537760 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:03.251745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:03.251700 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:03.351190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:03.351161 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:03.874915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:03.874873 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:04.072988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:04.072954 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:04.134352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:04.134268 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n"] Nov 04 00:39:04.137345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:04.137316 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-849709d8-l4c7n"] Nov 04 00:39:04.169104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:04.169019 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:04.644258 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:04.644216 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:05.293044 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:05.293009 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172c5038-00d7-4378-b056-3fe07ec00b16" path="/var/lib/kubelet/pods/172c5038-00d7-4378-b056-3fe07ec00b16/volumes" Nov 04 00:39:05.704743 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:39:05.704663 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:39:09.632825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:09.632787 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:10.641714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:10.641678 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:10.745762 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:10.745732 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:12.805920 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:12.805882 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:12.907205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:12.907167 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:33.247246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247208 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf"] Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247573 6208 cpu_manager.go:414] "containerMap: removing container" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerName="cleanup" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247584 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerName="cleanup" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247597 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247602 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfd5078-0143-4c64-8857-e76da9d546de" containerName="istio-proxy" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247611 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerName="cleanup" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247616 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerName="cleanup" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247626 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247630 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5979d19-2211-458a-9734-0a9a8ddfd617" containerName="istio-proxy" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247639 6208 cpu_manager.go:414] "containerMap: removing container" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerName="cleanup" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247644 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbe7e04-0cc5-41fc-ba74-51c8253fc0af" containerName="cleanup" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247650 6208 cpu_manager.go:414] "containerMap: removing container" podUID="172c5038-00d7-4378-b056-3fe07ec00b16" containerName="revoke-keys" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247654 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172c5038-00d7-4378-b056-3fe07ec00b16" containerName="revoke-keys" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247668 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerName="cleanup" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247672 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f24a17c-4f5d-461d-a1a1-c08bbd170078" containerName="cleanup" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247727 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8ed697-4cb5-49c9-b618-fbc6f91c7b0f" containerName="istio-proxy" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247734 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb1cff7-28c2-40e7-8dda-da12eb866ca6" containerName="cleanup" Nov 04 00:39:33.247734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247742 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="9946a8b2-7f05-48c3-9818-4badb91f171e" containerName="cleanup" Nov 04 00:39:33.248290 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247749 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="418b6385-c834-4532-94d5-e65086de1352" containerName="istio-proxy" Nov 04 00:39:33.248290 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247755 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="89632be3-5d3a-4b2f-b51c-7f6b21395b5a" containerName="cleanup" Nov 04 00:39:33.248290 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.247761 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="172c5038-00d7-4378-b056-3fe07ec00b16" containerName="revoke-keys" Nov 04 00:39:33.251058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.251017 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.258074 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.258019 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:39:33.260611 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.260585 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf"] Nov 04 00:39:33.260782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.260679 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:39:33.287163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.287137 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:39:33.287163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.287160 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:39:33.423961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.423927 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xxnnf\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.424168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.423968 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp7w2\" (UniqueName: \"kubernetes.io/projected/2db508a9-460f-4db6-a334-5ed4e302ee0a-kube-api-access-lp7w2\") pod \"maas-api-revoke-sub-85237e45-xxnnf\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.424239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.424162 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xxnnf\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.525083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.524987 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp7w2\" (UniqueName: \"kubernetes.io/projected/2db508a9-460f-4db6-a334-5ed4e302ee0a-kube-api-access-lp7w2\") pod \"maas-api-revoke-sub-85237e45-xxnnf\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.525083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.525082 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xxnnf\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.525285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.525139 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xxnnf\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.525942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.525919 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xxnnf\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.527446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.527427 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xxnnf\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.537753 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.537728 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp7w2\" (UniqueName: \"kubernetes.io/projected/2db508a9-460f-4db6-a334-5ed4e302ee0a-kube-api-access-lp7w2\") pod \"maas-api-revoke-sub-85237e45-xxnnf\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.564899 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.564875 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:33.842131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:33.842095 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf"] Nov 04 00:39:33.844238 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:39:33.844208 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db508a9_460f_4db6_a334_5ed4e302ee0a.slice/crio-f221af6e7e05c6c92a057ada8bca09ec95fa51f2e737b2cb607f7464056b54ec WatchSource:0}: Error finding container f221af6e7e05c6c92a057ada8bca09ec95fa51f2e737b2cb607f7464056b54ec: Status 404 returned error can't find the container with id f221af6e7e05c6c92a057ada8bca09ec95fa51f2e737b2cb607f7464056b54ec Nov 04 00:39:34.769890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:34.769862 6208 generic.go:356] "Generic (PLEG): container finished" podID="2db508a9-460f-4db6-a334-5ed4e302ee0a" containerID="1c0192c5e57d120da7c14a958501e15d7a685fb662d946fa94f33df377e24323" exitCode=0 Nov 04 00:39:34.770314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:34.769938 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" event={"ID":"2db508a9-460f-4db6-a334-5ed4e302ee0a","Type":"ContainerDied","Data":"1c0192c5e57d120da7c14a958501e15d7a685fb662d946fa94f33df377e24323"} Nov 04 00:39:34.770314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:34.769985 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" event={"ID":"2db508a9-460f-4db6-a334-5ed4e302ee0a","Type":"ContainerStarted","Data":"f221af6e7e05c6c92a057ada8bca09ec95fa51f2e737b2cb607f7464056b54ec"} Nov 04 00:39:35.707384 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:39:35.707346 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:39:35.911450 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:35.911427 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:36.047827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.047745 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db508a9-460f-4db6-a334-5ed4e302ee0a-kube-api-access-lp7w2\" (UniqueName: \"kubernetes.io/projected/2db508a9-460f-4db6-a334-5ed4e302ee0a-kube-api-access-lp7w2\") pod \"2db508a9-460f-4db6-a334-5ed4e302ee0a\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " Nov 04 00:39:36.048401 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.047848 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-xks-ca\") pod \"2db508a9-460f-4db6-a334-5ed4e302ee0a\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " Nov 04 00:39:36.048401 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.047924 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-openshift-ca\") pod \"2db508a9-460f-4db6-a334-5ed4e302ee0a\" (UID: \"2db508a9-460f-4db6-a334-5ed4e302ee0a\") " Nov 04 00:39:36.048401 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.048268 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-openshift-ca" pod "2db508a9-460f-4db6-a334-5ed4e302ee0a" (UID: "2db508a9-460f-4db6-a334-5ed4e302ee0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:39:36.049859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.049831 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-xks-ca" pod "2db508a9-460f-4db6-a334-5ed4e302ee0a" (UID: "2db508a9-460f-4db6-a334-5ed4e302ee0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:39:36.049859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.049840 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db508a9-460f-4db6-a334-5ed4e302ee0a-kube-api-access-lp7w2" pod "2db508a9-460f-4db6-a334-5ed4e302ee0a" (UID: "2db508a9-460f-4db6-a334-5ed4e302ee0a"). InnerVolumeSpecName "kube-api-access-lp7w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:39:36.149098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.149064 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:39:36.149098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.149094 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp7w2\" (UniqueName: \"kubernetes.io/projected/2db508a9-460f-4db6-a334-5ed4e302ee0a-kube-api-access-lp7w2\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:39:36.149098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.149105 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db508a9-460f-4db6-a334-5ed4e302ee0a-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:39:36.780671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.780632 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" event={"ID":"2db508a9-460f-4db6-a334-5ed4e302ee0a","Type":"ContainerDied","Data":"f221af6e7e05c6c92a057ada8bca09ec95fa51f2e737b2cb607f7464056b54ec"} Nov 04 00:39:36.780858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.780692 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f221af6e7e05c6c92a057ada8bca09ec95fa51f2e737b2cb607f7464056b54ec" Nov 04 00:39:36.780858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:36.780645 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf" Nov 04 00:39:37.400698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:37.400664 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:37.497355 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:37.497320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:43.061854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:43.061815 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:39:43.066812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:43.066789 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:39:43.746293 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:43.746252 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:43.845375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:43.845337 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:44.805755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:44.805718 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:48.012656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:48.012622 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:39:58.176944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:39:58.176903 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:40:03.122132 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:03.122062 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:40:03.224616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:03.224575 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:40:04.705115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:04.705083 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:40:05.703943 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:40:05.703904 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:40:13.340821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.340781 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw"] Nov 04 00:40:13.341239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.341188 6208 cpu_manager.go:414] "containerMap: removing container" podUID="2db508a9-460f-4db6-a334-5ed4e302ee0a" containerName="revoke-keys" Nov 04 00:40:13.341239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.341200 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db508a9-460f-4db6-a334-5ed4e302ee0a" containerName="revoke-keys" Nov 04 00:40:13.341317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.341279 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db508a9-460f-4db6-a334-5ed4e302ee0a" containerName="revoke-keys" Nov 04 00:40:13.344515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.344491 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.351401 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.351375 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:40:13.352762 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.352741 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:40:13.352988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.352958 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:40:13.353140 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.353088 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:40:13.353376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.353348 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw"] Nov 04 00:40:13.446669 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.446638 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-pjzhw\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.446844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.446702 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x787z\" (UniqueName: \"kubernetes.io/projected/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-kube-api-access-x787z\") pod \"maas-api-revoke-sub-f904bff9-pjzhw\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.446895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.446840 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-pjzhw\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.547449 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.547406 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-pjzhw\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.547615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.547481 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-pjzhw\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.547615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.547554 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x787z\" (UniqueName: \"kubernetes.io/projected/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-kube-api-access-x787z\") pod \"maas-api-revoke-sub-f904bff9-pjzhw\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.548177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.548156 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-pjzhw\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.549829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.549800 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-pjzhw\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.559714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.559689 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x787z\" (UniqueName: \"kubernetes.io/projected/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-kube-api-access-x787z\") pod \"maas-api-revoke-sub-f904bff9-pjzhw\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.658590 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.658524 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:13.933653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.933626 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw"] Nov 04 00:40:13.936107 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:40:13.936075 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2112d05_ca0d_4e9b_819e_66767cf9c2e4.slice/crio-382a2361f5afa761e3e23782ebcf263bc29fefb78f0c1502c370f42e5d159f28 WatchSource:0}: Error finding container 382a2361f5afa761e3e23782ebcf263bc29fefb78f0c1502c370f42e5d159f28: Status 404 returned error can't find the container with id 382a2361f5afa761e3e23782ebcf263bc29fefb78f0c1502c370f42e5d159f28 Nov 04 00:40:13.980361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:13.980332 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" event={"ID":"e2112d05-ca0d-4e9b-819e-66767cf9c2e4","Type":"ContainerStarted","Data":"382a2361f5afa761e3e23782ebcf263bc29fefb78f0c1502c370f42e5d159f28"} Nov 04 00:40:14.024056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:14.024005 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:40:14.986941 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:14.986901 6208 generic.go:356] "Generic (PLEG): container finished" podID="e2112d05-ca0d-4e9b-819e-66767cf9c2e4" containerID="d8fc3acb4c0645c73c8efcc184364838577ecb6f5f38023d1a464b0230f485fd" exitCode=0 Nov 04 00:40:14.987422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:14.986980 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" event={"ID":"e2112d05-ca0d-4e9b-819e-66767cf9c2e4","Type":"ContainerDied","Data":"d8fc3acb4c0645c73c8efcc184364838577ecb6f5f38023d1a464b0230f485fd"} Nov 04 00:40:16.127563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.127529 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:16.273993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.273895 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-openshift-ca\") pod \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " Nov 04 00:40:16.273993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.273942 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-xks-ca\") pod \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " Nov 04 00:40:16.274285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.274014 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-kube-api-access-x787z\" (UniqueName: \"kubernetes.io/projected/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-kube-api-access-x787z\") pod \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\" (UID: \"e2112d05-ca0d-4e9b-819e-66767cf9c2e4\") " Nov 04 00:40:16.274285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.274265 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-openshift-ca" pod "e2112d05-ca0d-4e9b-819e-66767cf9c2e4" (UID: "e2112d05-ca0d-4e9b-819e-66767cf9c2e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:40:16.274428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.274403 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:16.276248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.276214 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-xks-ca" pod "e2112d05-ca0d-4e9b-819e-66767cf9c2e4" (UID: "e2112d05-ca0d-4e9b-819e-66767cf9c2e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:40:16.276377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.276272 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-kube-api-access-x787z" pod "e2112d05-ca0d-4e9b-819e-66767cf9c2e4" (UID: "e2112d05-ca0d-4e9b-819e-66767cf9c2e4"). InnerVolumeSpecName "kube-api-access-x787z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:40:16.375627 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.375584 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:16.375627 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.375621 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x787z\" (UniqueName: \"kubernetes.io/projected/e2112d05-ca0d-4e9b-819e-66767cf9c2e4-kube-api-access-x787z\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:16.999102 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.999066 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" event={"ID":"e2112d05-ca0d-4e9b-819e-66767cf9c2e4","Type":"ContainerDied","Data":"382a2361f5afa761e3e23782ebcf263bc29fefb78f0c1502c370f42e5d159f28"} Nov 04 00:40:16.999312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.999152 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="382a2361f5afa761e3e23782ebcf263bc29fefb78f0c1502c370f42e5d159f28" Nov 04 00:40:16.999312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:16.999079 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw" Nov 04 00:40:35.722583 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:40:35.722479 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:40:36.752857 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.752819 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2"] Nov 04 00:40:36.753338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.753268 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e2112d05-ca0d-4e9b-819e-66767cf9c2e4" containerName="revoke-keys" Nov 04 00:40:36.753338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.753286 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2112d05-ca0d-4e9b-819e-66767cf9c2e4" containerName="revoke-keys" Nov 04 00:40:36.753419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.753383 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2112d05-ca0d-4e9b-819e-66767cf9c2e4" containerName="revoke-keys" Nov 04 00:40:36.756418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.756400 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:36.763862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.763043 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:40:36.763862 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.763077 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:40:36.764596 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.764572 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:40:36.764761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.764657 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:40:36.767515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.767489 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2"] Nov 04 00:40:36.834408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.834364 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mxw\" (UniqueName: \"kubernetes.io/projected/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-kube-api-access-95mxw\") pod \"maas-api-revoke-sub-f03629db-gltm2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:36.834606 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.834418 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gltm2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:36.834606 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.834557 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gltm2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:36.935487 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.935449 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mxw\" (UniqueName: \"kubernetes.io/projected/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-kube-api-access-95mxw\") pod \"maas-api-revoke-sub-f03629db-gltm2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:36.935665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.935498 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gltm2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:36.935665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.935541 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gltm2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:36.936291 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.936264 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gltm2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:36.937832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.937810 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gltm2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:36.948461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:36.948436 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mxw\" (UniqueName: \"kubernetes.io/projected/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-kube-api-access-95mxw\") pod \"maas-api-revoke-sub-f03629db-gltm2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:37.070389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:37.070316 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:37.345522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:37.345493 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2"] Nov 04 00:40:37.347707 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:40:37.347675 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6c66e2_c1aa_4230_ba7d_dc61b26086d2.slice/crio-4517025e002c8be451922e96bfd488c743f74d36e42080b64703b2e5c93f254c WatchSource:0}: Error finding container 4517025e002c8be451922e96bfd488c743f74d36e42080b64703b2e5c93f254c: Status 404 returned error can't find the container with id 4517025e002c8be451922e96bfd488c743f74d36e42080b64703b2e5c93f254c Nov 04 00:40:38.137106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:38.137066 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" event={"ID":"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2","Type":"ContainerStarted","Data":"64f8aaec3be9a41c6c80c8ce77ef717636d52c7fa0de5d99486c65995bf42002"} Nov 04 00:40:38.137605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:38.137128 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" event={"ID":"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2","Type":"ContainerStarted","Data":"4517025e002c8be451922e96bfd488c743f74d36e42080b64703b2e5c93f254c"} Nov 04 00:40:38.160775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:38.160727 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" podStartSLOduration=2.160713166 podStartE2EDuration="2.160713166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:40:38.159455083 +0000 UTC m=+4256.982051381" watchObservedRunningTime="2026-11-04 00:40:38.160713166 +0000 UTC m=+4256.983309329" Nov 04 00:40:39.148252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:39.148220 6208 generic.go:356] "Generic (PLEG): container finished" podID="4c6c66e2-c1aa-4230-ba7d-dc61b26086d2" containerID="64f8aaec3be9a41c6c80c8ce77ef717636d52c7fa0de5d99486c65995bf42002" exitCode=0 Nov 04 00:40:39.148657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:39.148310 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" event={"ID":"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2","Type":"ContainerDied","Data":"64f8aaec3be9a41c6c80c8ce77ef717636d52c7fa0de5d99486c65995bf42002"} Nov 04 00:40:40.288223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.288195 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:40.372599 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.372572 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-xks-ca\") pod \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " Nov 04 00:40:40.372752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.372629 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-openshift-ca\") pod \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " Nov 04 00:40:40.372752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.372715 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-kube-api-access-95mxw\" (UniqueName: \"kubernetes.io/projected/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-kube-api-access-95mxw\") pod \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\" (UID: \"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2\") " Nov 04 00:40:40.373062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.373005 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-openshift-ca" pod "4c6c66e2-c1aa-4230-ba7d-dc61b26086d2" (UID: "4c6c66e2-c1aa-4230-ba7d-dc61b26086d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:40:40.374804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.374781 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-xks-ca" pod "4c6c66e2-c1aa-4230-ba7d-dc61b26086d2" (UID: "4c6c66e2-c1aa-4230-ba7d-dc61b26086d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:40:40.374804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.374789 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-kube-api-access-95mxw" pod "4c6c66e2-c1aa-4230-ba7d-dc61b26086d2" (UID: "4c6c66e2-c1aa-4230-ba7d-dc61b26086d2"). InnerVolumeSpecName "kube-api-access-95mxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:40:40.473807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.473780 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95mxw\" (UniqueName: \"kubernetes.io/projected/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-kube-api-access-95mxw\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:40.473807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.473806 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:40.473971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:40.473817 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:41.159620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:41.159590 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" Nov 04 00:40:41.159795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:41.159593 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2" event={"ID":"4c6c66e2-c1aa-4230-ba7d-dc61b26086d2","Type":"ContainerDied","Data":"4517025e002c8be451922e96bfd488c743f74d36e42080b64703b2e5c93f254c"} Nov 04 00:40:41.159795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:41.159710 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4517025e002c8be451922e96bfd488c743f74d36e42080b64703b2e5c93f254c" Nov 04 00:40:53.022853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.022811 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg"] Nov 04 00:40:53.023306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.023271 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4c6c66e2-c1aa-4230-ba7d-dc61b26086d2" containerName="revoke-keys" Nov 04 00:40:53.023306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.023288 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6c66e2-c1aa-4230-ba7d-dc61b26086d2" containerName="revoke-keys" Nov 04 00:40:53.023418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.023380 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6c66e2-c1aa-4230-ba7d-dc61b26086d2" containerName="revoke-keys" Nov 04 00:40:53.026670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.026648 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.035628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.035396 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:40:53.035628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.035473 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:40:53.035628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.035517 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:40:53.035628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.035594 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:40:53.038510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.038477 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg"] Nov 04 00:40:53.120683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.120647 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4t2rg\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.120874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.120708 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq92x\" (UniqueName: \"kubernetes.io/projected/a71493ff-e4e2-4948-a1c0-34fcc30b663d-kube-api-access-kq92x\") pod \"maas-api-revoke-sub-0165d7d2-4t2rg\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.120874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.120841 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4t2rg\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.221758 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.221714 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4t2rg\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.221966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.221845 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4t2rg\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.221966 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.221902 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq92x\" (UniqueName: \"kubernetes.io/projected/a71493ff-e4e2-4948-a1c0-34fcc30b663d-kube-api-access-kq92x\") pod \"maas-api-revoke-sub-0165d7d2-4t2rg\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.222514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.222490 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4t2rg\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.224156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.224132 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4t2rg\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.235481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.235457 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq92x\" (UniqueName: \"kubernetes.io/projected/a71493ff-e4e2-4948-a1c0-34fcc30b663d-kube-api-access-kq92x\") pod \"maas-api-revoke-sub-0165d7d2-4t2rg\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.343980 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.343892 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:53.619343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:53.619314 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg"] Nov 04 00:40:53.621551 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:40:53.621521 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71493ff_e4e2_4948_a1c0_34fcc30b663d.slice/crio-6b046aa0bd5787f320317fe4bfd8f3d29f343cc591fd61675a3319f8740420d6 WatchSource:0}: Error finding container 6b046aa0bd5787f320317fe4bfd8f3d29f343cc591fd61675a3319f8740420d6: Status 404 returned error can't find the container with id 6b046aa0bd5787f320317fe4bfd8f3d29f343cc591fd61675a3319f8740420d6 Nov 04 00:40:54.244380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:54.244341 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" event={"ID":"a71493ff-e4e2-4948-a1c0-34fcc30b663d","Type":"ContainerStarted","Data":"a073fa164ab5511051dfa68e46c94557b9cc3089c90a1b4027069ff50b45ffaf"} Nov 04 00:40:54.244774 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:54.244402 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" event={"ID":"a71493ff-e4e2-4948-a1c0-34fcc30b663d","Type":"ContainerStarted","Data":"6b046aa0bd5787f320317fe4bfd8f3d29f343cc591fd61675a3319f8740420d6"} Nov 04 00:40:54.265315 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:54.265247 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" podStartSLOduration=1.265233936 podStartE2EDuration="1.265233936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:40:54.264248765 +0000 UTC m=+4273.086844929" watchObservedRunningTime="2026-11-04 00:40:54.265233936 +0000 UTC m=+4273.087830097" Nov 04 00:40:55.250403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:55.250369 6208 generic.go:356] "Generic (PLEG): container finished" podID="a71493ff-e4e2-4948-a1c0-34fcc30b663d" containerID="a073fa164ab5511051dfa68e46c94557b9cc3089c90a1b4027069ff50b45ffaf" exitCode=0 Nov 04 00:40:55.250912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:55.250460 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" event={"ID":"a71493ff-e4e2-4948-a1c0-34fcc30b663d","Type":"ContainerDied","Data":"a073fa164ab5511051dfa68e46c94557b9cc3089c90a1b4027069ff50b45ffaf"} Nov 04 00:40:56.389699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.389673 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:56.555305 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.555206 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-openshift-ca\") pod \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " Nov 04 00:40:56.555475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.555318 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71493ff-e4e2-4948-a1c0-34fcc30b663d-kube-api-access-kq92x\" (UniqueName: \"kubernetes.io/projected/a71493ff-e4e2-4948-a1c0-34fcc30b663d-kube-api-access-kq92x\") pod \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " Nov 04 00:40:56.555475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.555351 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-xks-ca\") pod \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\" (UID: \"a71493ff-e4e2-4948-a1c0-34fcc30b663d\") " Nov 04 00:40:56.555656 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.555625 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-openshift-ca" pod "a71493ff-e4e2-4948-a1c0-34fcc30b663d" (UID: "a71493ff-e4e2-4948-a1c0-34fcc30b663d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:40:56.557439 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.557412 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-xks-ca" pod "a71493ff-e4e2-4948-a1c0-34fcc30b663d" (UID: "a71493ff-e4e2-4948-a1c0-34fcc30b663d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:40:56.557532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.557446 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71493ff-e4e2-4948-a1c0-34fcc30b663d-kube-api-access-kq92x" pod "a71493ff-e4e2-4948-a1c0-34fcc30b663d" (UID: "a71493ff-e4e2-4948-a1c0-34fcc30b663d"). InnerVolumeSpecName "kube-api-access-kq92x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:40:56.656304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.656253 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq92x\" (UniqueName: \"kubernetes.io/projected/a71493ff-e4e2-4948-a1c0-34fcc30b663d-kube-api-access-kq92x\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:56.656304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.656300 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:56.656304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:56.656313 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71493ff-e4e2-4948-a1c0-34fcc30b663d-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:57.262172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:57.262135 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" Nov 04 00:40:57.262172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:57.262160 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg" event={"ID":"a71493ff-e4e2-4948-a1c0-34fcc30b663d","Type":"ContainerDied","Data":"6b046aa0bd5787f320317fe4bfd8f3d29f343cc591fd61675a3319f8740420d6"} Nov 04 00:40:57.264414 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:57.262211 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b046aa0bd5787f320317fe4bfd8f3d29f343cc591fd61675a3319f8740420d6" Nov 04 00:40:58.232454 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:58.232416 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:40:59.139427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.139387 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8"] Nov 04 00:40:59.139631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.139600 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" podUID="b086dba5-92ce-444f-b264-38744cca67f5" containerName="discovery" containerID="cri-o://0af1ace1c32fad9225368d86714d42bb01e2b13d9e36df0d13deaf8d6845fdeb" gracePeriod=30 Nov 04 00:40:59.275990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.275958 6208 generic.go:356] "Generic (PLEG): container finished" podID="b086dba5-92ce-444f-b264-38744cca67f5" containerID="0af1ace1c32fad9225368d86714d42bb01e2b13d9e36df0d13deaf8d6845fdeb" exitCode=0 Nov 04 00:40:59.276386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.276046 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" event={"ID":"b086dba5-92ce-444f-b264-38744cca67f5","Type":"ContainerDied","Data":"0af1ace1c32fad9225368d86714d42bb01e2b13d9e36df0d13deaf8d6845fdeb"} Nov 04 00:40:59.399526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.399468 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 04 00:40:59.479595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.479560 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-istio-token\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-istio-token\") pod \"b086dba5-92ce-444f-b264-38744cca67f5\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " Nov 04 00:40:59.479795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.479599 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-cacerts\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-cacerts\") pod \"b086dba5-92ce-444f-b264-38744cca67f5\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " Nov 04 00:40:59.479795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.479620 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-kubeconfig\") pod \"b086dba5-92ce-444f-b264-38744cca67f5\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " Nov 04 00:40:59.479795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.479653 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-ca-configmap\") pod \"b086dba5-92ce-444f-b264-38744cca67f5\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " Nov 04 00:40:59.479795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.479722 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-kube-api-access-6rw6r\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-kube-api-access-6rw6r\") pod \"b086dba5-92ce-444f-b264-38744cca67f5\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " Nov 04 00:40:59.480065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.479850 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b086dba5-92ce-444f-b264-38744cca67f5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b086dba5-92ce-444f-b264-38744cca67f5-local-certs\") pod \"b086dba5-92ce-444f-b264-38744cca67f5\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " Nov 04 00:40:59.480065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.479901 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-dns-cert\") pod \"b086dba5-92ce-444f-b264-38744cca67f5\" (UID: \"b086dba5-92ce-444f-b264-38744cca67f5\") " Nov 04 00:40:59.480218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.480187 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-ca-configmap" pod "b086dba5-92ce-444f-b264-38744cca67f5" (UID: "b086dba5-92ce-444f-b264-38744cca67f5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:40:59.480303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.480280 6208 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-ca-configmap\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:59.482148 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.482105 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-dns-cert" pod "b086dba5-92ce-444f-b264-38744cca67f5" (UID: "b086dba5-92ce-444f-b264-38744cca67f5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:40:59.482584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.482557 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-cacerts" pod "b086dba5-92ce-444f-b264-38744cca67f5" (UID: "b086dba5-92ce-444f-b264-38744cca67f5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:40:59.482584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.482566 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-kube-api-access-6rw6r" pod "b086dba5-92ce-444f-b264-38744cca67f5" (UID: "b086dba5-92ce-444f-b264-38744cca67f5"). InnerVolumeSpecName "kube-api-access-6rw6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:40:59.482715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.482563 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b086dba5-92ce-444f-b264-38744cca67f5-local-certs" pod "b086dba5-92ce-444f-b264-38744cca67f5" (UID: "b086dba5-92ce-444f-b264-38744cca67f5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:40:59.482952 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.482932 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-istio-token" pod "b086dba5-92ce-444f-b264-38744cca67f5" (UID: "b086dba5-92ce-444f-b264-38744cca67f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:40:59.482952 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.482938 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-kubeconfig" pod "b086dba5-92ce-444f-b264-38744cca67f5" (UID: "b086dba5-92ce-444f-b264-38744cca67f5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:40:59.581513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.581464 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rw6r\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-kube-api-access-6rw6r\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:59.581513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.581500 6208 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b086dba5-92ce-444f-b264-38744cca67f5-local-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:59.581513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.581515 6208 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-csr-dns-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:59.581782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.581527 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b086dba5-92ce-444f-b264-38744cca67f5-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:59.581782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.581536 6208 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-cacerts\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:40:59.581782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:40:59.581544 6208 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b086dba5-92ce-444f-b264-38744cca67f5-istio-kubeconfig\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:41:00.282892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:00.282861 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" Nov 04 00:41:00.282892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:00.282882 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8" event={"ID":"b086dba5-92ce-444f-b264-38744cca67f5","Type":"ContainerDied","Data":"df4d2bef45f6b2c73c96211cff2f3b6c7fd97e5f4126f9f6cff69ac121ec943a"} Nov 04 00:41:00.283450 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:00.282935 6208 scope.go:122] "RemoveContainer" containerID="0af1ace1c32fad9225368d86714d42bb01e2b13d9e36df0d13deaf8d6845fdeb" Nov 04 00:41:00.309501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:00.309472 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8"] Nov 04 00:41:00.314289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:00.314263 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-92wb8"] Nov 04 00:41:01.292940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:01.292907 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b086dba5-92ce-444f-b264-38744cca67f5" path="/var/lib/kubelet/pods/b086dba5-92ce-444f-b264-38744cca67f5/volumes" Nov 04 00:41:03.902604 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.902555 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69"] Nov 04 00:41:03.903430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.903401 6208 cpu_manager.go:414] "containerMap: removing container" podUID="b086dba5-92ce-444f-b264-38744cca67f5" containerName="discovery" Nov 04 00:41:03.903552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.903431 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b086dba5-92ce-444f-b264-38744cca67f5" containerName="discovery" Nov 04 00:41:03.903552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.903494 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a71493ff-e4e2-4948-a1c0-34fcc30b663d" containerName="revoke-keys" Nov 04 00:41:03.903552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.903503 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71493ff-e4e2-4948-a1c0-34fcc30b663d" containerName="revoke-keys" Nov 04 00:41:03.903713 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.903680 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71493ff-e4e2-4948-a1c0-34fcc30b663d" containerName="revoke-keys" Nov 04 00:41:03.903713 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.903704 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="b086dba5-92ce-444f-b264-38744cca67f5" containerName="discovery" Nov 04 00:41:03.908836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.908799 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:03.912450 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.912349 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69"] Nov 04 00:41:03.914190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.914166 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:41:03.914382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.914174 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:41:03.914382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.914238 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:41:03.915653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.915630 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:41:03.920160 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.920134 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-d9z69\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:03.920268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.920191 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-d9z69\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:03.920329 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:03.920317 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5mw\" (UniqueName: \"kubernetes.io/projected/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-kube-api-access-hq5mw\") pod \"maas-api-revoke-sub-6b19338f-d9z69\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:04.021531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:04.021489 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5mw\" (UniqueName: \"kubernetes.io/projected/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-kube-api-access-hq5mw\") pod \"maas-api-revoke-sub-6b19338f-d9z69\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:04.021752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:04.021572 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-d9z69\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:04.021752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:04.021637 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-d9z69\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:04.022417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:04.022388 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-d9z69\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:04.023988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:04.023969 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-d9z69\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:04.033620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:04.033595 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5mw\" (UniqueName: \"kubernetes.io/projected/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-kube-api-access-hq5mw\") pod \"maas-api-revoke-sub-6b19338f-d9z69\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:04.226193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:04.226157 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:04.501350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:04.501321 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69"] Nov 04 00:41:04.503377 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:41:04.503346 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407605d5_b8a1_488c_8d3a_7f9b5ad59a5a.slice/crio-981ecac312e3aeb7bfe23a6d05f49a1a71ddf885820a4d83383489f7f6dc484f WatchSource:0}: Error finding container 981ecac312e3aeb7bfe23a6d05f49a1a71ddf885820a4d83383489f7f6dc484f: Status 404 returned error can't find the container with id 981ecac312e3aeb7bfe23a6d05f49a1a71ddf885820a4d83383489f7f6dc484f Nov 04 00:41:04.763248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:04.763202 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:41:05.310792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:05.310761 6208 generic.go:356] "Generic (PLEG): container finished" podID="407605d5-b8a1-488c-8d3a-7f9b5ad59a5a" containerID="68481c80428c84a9a5ee0eb8ecb3eedb7142bda29d4af5a42a05a82959563da6" exitCode=0 Nov 04 00:41:05.311120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:05.310802 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" event={"ID":"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a","Type":"ContainerDied","Data":"68481c80428c84a9a5ee0eb8ecb3eedb7142bda29d4af5a42a05a82959563da6"} Nov 04 00:41:05.311120 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:05.310846 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" event={"ID":"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a","Type":"ContainerStarted","Data":"981ecac312e3aeb7bfe23a6d05f49a1a71ddf885820a4d83383489f7f6dc484f"} Nov 04 00:41:05.704053 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:41:05.703956 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:41:06.446779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.446753 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:06.543855 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.543818 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-xks-ca\") pod \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " Nov 04 00:41:06.544099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.543920 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-kube-api-access-hq5mw\" (UniqueName: \"kubernetes.io/projected/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-kube-api-access-hq5mw\") pod \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " Nov 04 00:41:06.544099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.543957 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-openshift-ca\") pod \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\" (UID: \"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a\") " Nov 04 00:41:06.544375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.544351 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-openshift-ca" pod "407605d5-b8a1-488c-8d3a-7f9b5ad59a5a" (UID: "407605d5-b8a1-488c-8d3a-7f9b5ad59a5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:41:06.545969 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.545941 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-xks-ca" pod "407605d5-b8a1-488c-8d3a-7f9b5ad59a5a" (UID: "407605d5-b8a1-488c-8d3a-7f9b5ad59a5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:41:06.546119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.546000 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-kube-api-access-hq5mw" pod "407605d5-b8a1-488c-8d3a-7f9b5ad59a5a" (UID: "407605d5-b8a1-488c-8d3a-7f9b5ad59a5a"). InnerVolumeSpecName "kube-api-access-hq5mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:41:06.645546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.645466 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq5mw\" (UniqueName: \"kubernetes.io/projected/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-kube-api-access-hq5mw\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:41:06.645546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.645505 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:41:06.645546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:06.645521 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:41:07.322124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:07.322098 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" Nov 04 00:41:07.322360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:07.322099 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69" event={"ID":"407605d5-b8a1-488c-8d3a-7f9b5ad59a5a","Type":"ContainerDied","Data":"981ecac312e3aeb7bfe23a6d05f49a1a71ddf885820a4d83383489f7f6dc484f"} Nov 04 00:41:07.322360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:07.322229 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="981ecac312e3aeb7bfe23a6d05f49a1a71ddf885820a4d83383489f7f6dc484f" Nov 04 00:41:14.084332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:14.084284 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:41:17.146901 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.146861 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq"] Nov 04 00:41:17.149163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.149131 6208 cpu_manager.go:414] "containerMap: removing container" podUID="407605d5-b8a1-488c-8d3a-7f9b5ad59a5a" containerName="revoke-keys" Nov 04 00:41:17.149331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.149315 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407605d5-b8a1-488c-8d3a-7f9b5ad59a5a" containerName="revoke-keys" Nov 04 00:41:17.149698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.149682 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="407605d5-b8a1-488c-8d3a-7f9b5ad59a5a" containerName="revoke-keys" Nov 04 00:41:17.159294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.159268 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.183343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.181290 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq"] Nov 04 00:41:17.197106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.197073 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4fbc73a8-kserve-self-signed-certs\"" Nov 04 00:41:17.203181 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.203058 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:41:17.248931 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.248887 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-model-cache\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.248931 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.248937 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-home\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.249146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.249015 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-dshm\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.249146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.249113 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tls-certs\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.249222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.249178 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt5sr\" (UniqueName: \"kubernetes.io/projected/1fe45469-94e6-4323-b8f9-c21f90cd55f2-kube-api-access-rt5sr\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.249222 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.249204 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tmp-dir\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.350052 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.349987 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-dshm\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.350246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.350089 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tls-certs\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.350246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.350154 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt5sr\" (UniqueName: \"kubernetes.io/projected/1fe45469-94e6-4323-b8f9-c21f90cd55f2-kube-api-access-rt5sr\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.350246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.350194 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tmp-dir\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.350429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.350248 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-model-cache\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.350429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.350287 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-home\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.350613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.350593 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-home\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.350687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.350642 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tmp-dir\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.350687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.350671 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-model-cache\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.352915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.352889 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-dshm\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.353191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.353173 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tls-certs\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.362577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.362557 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt5sr\" (UniqueName: \"kubernetes.io/projected/1fe45469-94e6-4323-b8f9-c21f90cd55f2-kube-api-access-rt5sr\") pod \"e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.478373 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.478337 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh"] Nov 04 00:41:17.481719 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.481697 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:17.495177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.491484 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4fbc73a8-epp-sa-dockercfg-knbng\"" Nov 04 00:41:17.495835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.495792 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh"] Nov 04 00:41:17.511079 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.511028 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:17.531957 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.531928 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:41:17.552789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.552284 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236631df-0017-493b-9bc0-05af8c75c911-tls-certs\") pod \"e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh\" (UID: \"236631df-0017-493b-9bc0-05af8c75c911\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:17.552789 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.552364 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnbfr\" (UniqueName: \"kubernetes.io/projected/236631df-0017-493b-9bc0-05af8c75c911-kube-api-access-nnbfr\") pod \"e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh\" (UID: \"236631df-0017-493b-9bc0-05af8c75c911\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:17.653769 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.653724 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnbfr\" (UniqueName: \"kubernetes.io/projected/236631df-0017-493b-9bc0-05af8c75c911-kube-api-access-nnbfr\") pod \"e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh\" (UID: \"236631df-0017-493b-9bc0-05af8c75c911\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:17.653971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.653876 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236631df-0017-493b-9bc0-05af8c75c911-tls-certs\") pod \"e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh\" (UID: \"236631df-0017-493b-9bc0-05af8c75c911\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:17.656615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.656588 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236631df-0017-493b-9bc0-05af8c75c911-tls-certs\") pod \"e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh\" (UID: \"236631df-0017-493b-9bc0-05af8c75c911\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:17.669186 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.669134 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnbfr\" (UniqueName: \"kubernetes.io/projected/236631df-0017-493b-9bc0-05af8c75c911-kube-api-access-nnbfr\") pod \"e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh\" (UID: \"236631df-0017-493b-9bc0-05af8c75c911\") " pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:17.794117 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.794021 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq"] Nov 04 00:41:17.796712 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:41:17.796666 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe45469_94e6_4323_b8f9_c21f90cd55f2.slice/crio-1a55da9e3f85d09ec6a531349a59029d2f5fc672d433b94a7888d3d3196c75b3 WatchSource:0}: Error finding container 1a55da9e3f85d09ec6a531349a59029d2f5fc672d433b94a7888d3d3196c75b3: Status 404 returned error can't find the container with id 1a55da9e3f85d09ec6a531349a59029d2f5fc672d433b94a7888d3d3196c75b3 Nov 04 00:41:17.801179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:17.801159 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:18.084978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:18.084950 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh"] Nov 04 00:41:18.086676 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:41:18.086649 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236631df_0017_493b_9bc0_05af8c75c911.slice/crio-48c258827b614d9c53f72a2a27cb8894b743f85fed58e9ae9d0ddb483d68dd98 WatchSource:0}: Error finding container 48c258827b614d9c53f72a2a27cb8894b743f85fed58e9ae9d0ddb483d68dd98: Status 404 returned error can't find the container with id 48c258827b614d9c53f72a2a27cb8894b743f85fed58e9ae9d0ddb483d68dd98 Nov 04 00:41:18.089892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:18.089874 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:41:18.394026 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:18.393936 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" event={"ID":"236631df-0017-493b-9bc0-05af8c75c911","Type":"ContainerStarted","Data":"48c258827b614d9c53f72a2a27cb8894b743f85fed58e9ae9d0ddb483d68dd98"} Nov 04 00:41:18.395639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:18.395607 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" event={"ID":"1fe45469-94e6-4323-b8f9-c21f90cd55f2","Type":"ContainerStarted","Data":"93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536"} Nov 04 00:41:18.395769 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:18.395643 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" event={"ID":"1fe45469-94e6-4323-b8f9-c21f90cd55f2","Type":"ContainerStarted","Data":"1a55da9e3f85d09ec6a531349a59029d2f5fc672d433b94a7888d3d3196c75b3"} Nov 04 00:41:18.416392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:18.416336 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" podStartSLOduration=1.416320554 podStartE2EDuration="1.416320554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:41:18.415523549 +0000 UTC m=+4297.238119712" watchObservedRunningTime="2026-11-04 00:41:18.416320554 +0000 UTC m=+4297.238916720" Nov 04 00:41:18.512097 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:18.512024 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:22.683824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:22.683671 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:41:24.458620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:24.458581 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" event={"ID":"236631df-0017-493b-9bc0-05af8c75c911","Type":"ContainerStarted","Data":"d94b32513ff946dffc801aee69986a54f4597b7c0a9f54325979a726faa7c3b1"} Nov 04 00:41:24.459053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:24.458666 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:24.479493 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:24.479440 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" podStartSLOduration=2.117554351 podStartE2EDuration="7.479426691s" totalImagesPullingTime="5.36187234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:41:24.477212953 +0000 UTC m=+4303.299809114" watchObservedRunningTime="2026-11-04 00:41:24.479426691 +0000 UTC m=+4303.302022852" Nov 04 00:41:27.511632 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:27.511598 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:27.524103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:27.524074 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:28.492403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:28.492367 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:41:35.703309 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:41:35.703266 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:41:55.466001 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:55.465922 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:41:58.294451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:41:58.294412 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:42:04.824556 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:04.824508 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:42:05.704535 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:42:05.704492 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:42:14.143261 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:14.143221 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:42:33.975112 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:33.975072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:42:34.821446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:34.821400 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:42:35.706207 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:42:35.706170 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:42:38.644878 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:38.644837 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh"] Nov 04 00:42:38.657815 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:38.657777 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq"] Nov 04 00:42:38.661093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:38.661050 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:42:38.762109 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:38.762067 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:42:45.487058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:45.485125 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" podUID="236631df-0017-493b-9bc0-05af8c75c911" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:42:45.669056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:45.665166 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:42:45.669056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:45.665242 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:42:53.679347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.679289 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" podUID="1fe45469-94e6-4323-b8f9-c21f90cd55f2" containerName="main" containerID="cri-o://93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536" gracePeriod=45 Nov 04 00:42:53.685668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.685189 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" podUID="236631df-0017-493b-9bc0-05af8c75c911" containerName="main" containerID="cri-o://d94b32513ff946dffc801aee69986a54f4597b7c0a9f54325979a726faa7c3b1" gracePeriod=45 Nov 04 00:42:53.934397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.934337 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:42:53.976631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.976600 6208 generic.go:356] "Generic (PLEG): container finished" podID="1fe45469-94e6-4323-b8f9-c21f90cd55f2" containerID="93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536" exitCode=0 Nov 04 00:42:53.976799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.976641 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" Nov 04 00:42:53.976799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.976685 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" event={"ID":"1fe45469-94e6-4323-b8f9-c21f90cd55f2","Type":"ContainerDied","Data":"93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536"} Nov 04 00:42:53.976799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.976750 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq" event={"ID":"1fe45469-94e6-4323-b8f9-c21f90cd55f2","Type":"ContainerDied","Data":"1a55da9e3f85d09ec6a531349a59029d2f5fc672d433b94a7888d3d3196c75b3"} Nov 04 00:42:53.976799 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.976775 6208 scope.go:122] "RemoveContainer" containerID="93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536" Nov 04 00:42:53.988177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.988158 6208 scope.go:122] "RemoveContainer" containerID="93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536" Nov 04 00:42:53.988469 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:42:53.988442 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536\": container with ID starting with 93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536 not found: ID does not exist" containerID="93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536" Nov 04 00:42:53.988570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:53.988476 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536"} err="failed to get container status \"93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536\": rpc error: code = NotFound desc = could not find container \"93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536\": container with ID starting with 93f7d28787614ec7dedc9de0234f53c8db25ab2c9ecb4982697c49b89b33b536 not found: ID does not exist" Nov 04 00:42:54.000307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000288 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tmp-dir\") pod \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " Nov 04 00:42:54.000406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000319 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-dshm\") pod \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " Nov 04 00:42:54.000406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000361 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tls-certs\") pod \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " Nov 04 00:42:54.000406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000384 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-model-cache\") pod \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " Nov 04 00:42:54.000585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000413 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe45469-94e6-4323-b8f9-c21f90cd55f2-kube-api-access-rt5sr\" (UniqueName: \"kubernetes.io/projected/1fe45469-94e6-4323-b8f9-c21f90cd55f2-kube-api-access-rt5sr\") pod \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " Nov 04 00:42:54.000585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000440 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-home\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-home\") pod \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\" (UID: \"1fe45469-94e6-4323-b8f9-c21f90cd55f2\") " Nov 04 00:42:54.000585 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000550 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tmp-dir" pod "1fe45469-94e6-4323-b8f9-c21f90cd55f2" (UID: "1fe45469-94e6-4323-b8f9-c21f90cd55f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:42:54.000756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000643 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-model-cache" pod "1fe45469-94e6-4323-b8f9-c21f90cd55f2" (UID: "1fe45469-94e6-4323-b8f9-c21f90cd55f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:42:54.000812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000762 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-home" pod "1fe45469-94e6-4323-b8f9-c21f90cd55f2" (UID: "1fe45469-94e6-4323-b8f9-c21f90cd55f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:42:54.000889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000866 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:42:54.000889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000878 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:42:54.001002 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.000889 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:42:54.002680 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.002649 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tls-certs" pod "1fe45469-94e6-4323-b8f9-c21f90cd55f2" (UID: "1fe45469-94e6-4323-b8f9-c21f90cd55f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:42:54.002767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.002660 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-dshm" pod "1fe45469-94e6-4323-b8f9-c21f90cd55f2" (UID: "1fe45469-94e6-4323-b8f9-c21f90cd55f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:42:54.002767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.002723 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe45469-94e6-4323-b8f9-c21f90cd55f2-kube-api-access-rt5sr" pod "1fe45469-94e6-4323-b8f9-c21f90cd55f2" (UID: "1fe45469-94e6-4323-b8f9-c21f90cd55f2"). InnerVolumeSpecName "kube-api-access-rt5sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:42:54.102437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.102399 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe45469-94e6-4323-b8f9-c21f90cd55f2-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:42:54.102437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.102431 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe45469-94e6-4323-b8f9-c21f90cd55f2-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:42:54.102437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.102442 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt5sr\" (UniqueName: \"kubernetes.io/projected/1fe45469-94e6-4323-b8f9-c21f90cd55f2-kube-api-access-rt5sr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:42:54.302007 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.301967 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq"] Nov 04 00:42:54.307985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:54.307955 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-5f794b5fb5-vp5zq"] Nov 04 00:42:55.294062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:55.293997 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe45469-94e6-4323-b8f9-c21f90cd55f2" path="/var/lib/kubelet/pods/1fe45469-94e6-4323-b8f9-c21f90cd55f2/volumes" Nov 04 00:42:55.466612 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:55.466567 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" podUID="236631df-0017-493b-9bc0-05af8c75c911" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:42:58.353073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:42:58.353007 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:04.881692 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:04.881653 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:04.982090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:04.982057 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:05.407541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:05.407504 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:05.466729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:05.466693 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" podUID="236631df-0017-493b-9bc0-05af8c75c911" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:43:05.466919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:05.466795 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:43:05.515636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:05.515597 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:05.704266 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:43:05.704174 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:43:15.465988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:15.465945 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" podUID="236631df-0017-493b-9bc0-05af8c75c911" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:43:18.594497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:18.594453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:18.693233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:18.693196 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:19.790778 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:19.790694 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:19.956374 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:19.956334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:20.116836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:20.116749 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:20.840669 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:20.840629 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:21.642220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:21.642178 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:22.543362 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:22.543322 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:23.448297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:23.448260 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:24.143642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.143607 6208 generic.go:356] "Generic (PLEG): container finished" podID="236631df-0017-493b-9bc0-05af8c75c911" containerID="d94b32513ff946dffc801aee69986a54f4597b7c0a9f54325979a726faa7c3b1" exitCode=0 Nov 04 00:43:24.144065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.143653 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" event={"ID":"236631df-0017-493b-9bc0-05af8c75c911","Type":"ContainerDied","Data":"d94b32513ff946dffc801aee69986a54f4597b7c0a9f54325979a726faa7c3b1"} Nov 04 00:43:24.151184 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.151150 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:24.250546 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.250509 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:24.335268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.335243 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:43:24.406872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.406800 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/236631df-0017-493b-9bc0-05af8c75c911-tls-certs\" (UniqueName: \"kubernetes.io/secret/236631df-0017-493b-9bc0-05af8c75c911-tls-certs\") pod \"236631df-0017-493b-9bc0-05af8c75c911\" (UID: \"236631df-0017-493b-9bc0-05af8c75c911\") " Nov 04 00:43:24.407070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.406872 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236631df-0017-493b-9bc0-05af8c75c911-kube-api-access-nnbfr\" (UniqueName: \"kubernetes.io/projected/236631df-0017-493b-9bc0-05af8c75c911-kube-api-access-nnbfr\") pod \"236631df-0017-493b-9bc0-05af8c75c911\" (UID: \"236631df-0017-493b-9bc0-05af8c75c911\") " Nov 04 00:43:24.408832 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.408805 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236631df-0017-493b-9bc0-05af8c75c911-tls-certs" pod "236631df-0017-493b-9bc0-05af8c75c911" (UID: "236631df-0017-493b-9bc0-05af8c75c911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:43:24.408959 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.408941 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236631df-0017-493b-9bc0-05af8c75c911-kube-api-access-nnbfr" pod "236631df-0017-493b-9bc0-05af8c75c911" (UID: "236631df-0017-493b-9bc0-05af8c75c911"). InnerVolumeSpecName "kube-api-access-nnbfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:43:24.507516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.507483 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/236631df-0017-493b-9bc0-05af8c75c911-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:43:24.507516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.507514 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnbfr\" (UniqueName: \"kubernetes.io/projected/236631df-0017-493b-9bc0-05af8c75c911-kube-api-access-nnbfr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:43:24.931550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:24.931516 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:25.150050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:25.149995 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" event={"ID":"236631df-0017-493b-9bc0-05af8c75c911","Type":"ContainerDied","Data":"48c258827b614d9c53f72a2a27cb8894b743f85fed58e9ae9d0ddb483d68dd98"} Nov 04 00:43:25.150050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:25.150048 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh" Nov 04 00:43:25.150695 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:25.150072 6208 scope.go:122] "RemoveContainer" containerID="d94b32513ff946dffc801aee69986a54f4597b7c0a9f54325979a726faa7c3b1" Nov 04 00:43:25.180608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:25.180560 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh"] Nov 04 00:43:25.181437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:25.181405 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4fbc73a8-kserve-router-scheduler-78d4d76948-x6slh"] Nov 04 00:43:25.292728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:25.292693 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236631df-0017-493b-9bc0-05af8c75c911" path="/var/lib/kubelet/pods/236631df-0017-493b-9bc0-05af8c75c911/volumes" Nov 04 00:43:25.723712 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:25.723672 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:26.459973 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:26.459927 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:27.162790 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:27.162751 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:27.930792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:27.930750 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:28.710391 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:28.710350 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:29.545366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:29.545326 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:30.427819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:30.427786 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:31.150964 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:31.150900 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:31.932306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:31.932272 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:32.723481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:32.723443 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:33.440049 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:33.439993 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:33.659886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:33.659849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:34.320547 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:34.320508 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:34.437472 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:34.437436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:34.543317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:34.543278 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:35.348175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:35.348136 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:35.703751 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:43:35.703660 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:43:36.132209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.132174 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:36.236876 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.236838 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm"] Nov 04 00:43:36.237284 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.237269 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1fe45469-94e6-4323-b8f9-c21f90cd55f2" containerName="main" Nov 04 00:43:36.237367 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.237287 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe45469-94e6-4323-b8f9-c21f90cd55f2" containerName="main" Nov 04 00:43:36.237367 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.237314 6208 cpu_manager.go:414] "containerMap: removing container" podUID="236631df-0017-493b-9bc0-05af8c75c911" containerName="main" Nov 04 00:43:36.237367 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.237320 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236631df-0017-493b-9bc0-05af8c75c911" containerName="main" Nov 04 00:43:36.237602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.237383 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe45469-94e6-4323-b8f9-c21f90cd55f2" containerName="main" Nov 04 00:43:36.237602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.237398 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="236631df-0017-493b-9bc0-05af8c75c911" containerName="main" Nov 04 00:43:36.247970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.247931 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.250018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.249988 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm"] Nov 04 00:43:36.258487 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.258453 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ca4fbe\"/\"openshift-service-ca.crt\"" Nov 04 00:43:36.258642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.258512 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ca4fbe\"/\"kube-root-ca.crt\"" Nov 04 00:43:36.276802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.276770 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ca4fbe\"/\"simulator-single-main-ca4fbe-kserve-self-signed-certs\"" Nov 04 00:43:36.276802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.276796 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ca4fbe\"/\"default-dockercfg-sjh8n\"" Nov 04 00:43:36.287338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.287308 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:36.327829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.327785 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-home\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.327995 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.327904 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-model-cache\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.327995 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.327972 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-tmp-dir\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.328137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.327994 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-dshm\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.328137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.328083 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b1256b-1b2e-4c1a-9134-1ced4b096414-tls-certs\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.328137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.328112 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qsf\" (UniqueName: \"kubernetes.io/projected/25b1256b-1b2e-4c1a-9134-1ced4b096414-kube-api-access-65qsf\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.428724 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.428622 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-model-cache\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.428724 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.428669 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-tmp-dir\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.429317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.428782 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-dshm\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.429317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.428867 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b1256b-1b2e-4c1a-9134-1ced4b096414-tls-certs\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.429317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.428888 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qsf\" (UniqueName: \"kubernetes.io/projected/25b1256b-1b2e-4c1a-9134-1ced4b096414-kube-api-access-65qsf\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.429317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.428937 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-home\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.429317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.429021 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-tmp-dir\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.429317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.429094 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-model-cache\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.429317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.429288 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-home\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.431174 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.431149 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-dshm\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.431492 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.431472 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b1256b-1b2e-4c1a-9134-1ced4b096414-tls-certs\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.444354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.444282 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qsf\" (UniqueName: \"kubernetes.io/projected/25b1256b-1b2e-4c1a-9134-1ced4b096414-kube-api-access-65qsf\") pod \"simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.561274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.561230 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:36.838386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.838358 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm"] Nov 04 00:43:36.840655 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:43:36.840627 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25b1256b_1b2e_4c1a_9134_1ced4b096414.slice/crio-e3c6b090307414dfec01b87f5938a8cee5bee60b5c2b0c8cfdcf79b51f5201dc WatchSource:0}: Error finding container e3c6b090307414dfec01b87f5938a8cee5bee60b5c2b0c8cfdcf79b51f5201dc: Status 404 returned error can't find the container with id e3c6b090307414dfec01b87f5938a8cee5bee60b5c2b0c8cfdcf79b51f5201dc Nov 04 00:43:36.941361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:36.941326 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:37.219648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:37.219612 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" event={"ID":"25b1256b-1b2e-4c1a-9134-1ced4b096414","Type":"ContainerStarted","Data":"a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0"} Nov 04 00:43:37.219826 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:37.219655 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" event={"ID":"25b1256b-1b2e-4c1a-9134-1ced4b096414","Type":"ContainerStarted","Data":"e3c6b090307414dfec01b87f5938a8cee5bee60b5c2b0c8cfdcf79b51f5201dc"} Nov 04 00:43:37.242307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:37.242257 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" podStartSLOduration=1.242245395 podStartE2EDuration="1.242245395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:43:37.239244647 +0000 UTC m=+4436.061840809" watchObservedRunningTime="2026-11-04 00:43:37.242245395 +0000 UTC m=+4436.064841556" Nov 04 00:43:37.562344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:37.562260 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:38.840297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:38.840260 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:39.460652 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:39.460614 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:40.303179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:40.303130 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:41.143295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:41.143255 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:42.055803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:42.055761 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:42.703194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:42.703157 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:42.808287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:42.808213 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:42.908718 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:42.908682 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:43.630620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:43.630568 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:44.435610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:44.435574 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:45.230866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:45.230811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:46.029252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:46.029207 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:46.561596 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:46.561550 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:46.574003 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:46.573977 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:46.756548 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:46.756507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:47.283802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:47.283727 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:43:47.547427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:47.547338 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:48.338403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:48.338367 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:49.140437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:49.140402 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:49.933414 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:49.933367 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:50.845817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:50.845783 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:51.650951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:51.650914 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:52.457542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:52.457490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:53.245209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:53.245165 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:54.054947 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:54.054900 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:54.847892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:54.847841 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:55.815161 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:55.815129 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:56.748382 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:56.748342 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:57.639185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:57.639146 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:58.413480 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:58.413442 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:58.514183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:58.514148 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:43:59.355864 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:43:59.355833 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:00.126678 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:00.126638 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:00.930276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:00.930229 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:01.741257 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:01.741225 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:02.526537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:02.526483 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:03.259233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:03.259192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:04.024073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:04.024010 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:04.751445 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:04.751404 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:04.942107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:04.942067 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:05.605529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:05.605492 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:05.705597 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:44:05.705555 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:44:05.949951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:05.949865 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l"] Nov 04 00:44:05.953559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:05.953535 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:05.960751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:05.960461 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:44:05.960751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:05.960591 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:44:05.960751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:05.960645 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:44:05.960751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:05.960727 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:44:05.963487 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:05.963462 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l"] Nov 04 00:44:06.125575 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.125543 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4rp5\" (UniqueName: \"kubernetes.io/projected/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-kube-api-access-t4rp5\") pod \"maas-api-revoke-sub-a948a352-ljt7l\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.125764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.125586 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a948a352-ljt7l\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.125764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.125713 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a948a352-ljt7l\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.226863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.226829 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4rp5\" (UniqueName: \"kubernetes.io/projected/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-kube-api-access-t4rp5\") pod \"maas-api-revoke-sub-a948a352-ljt7l\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.226863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.226868 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a948a352-ljt7l\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.227173 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.226912 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a948a352-ljt7l\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.227552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.227532 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a948a352-ljt7l\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.229402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.229373 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a948a352-ljt7l\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.239047 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.239012 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4rp5\" (UniqueName: \"kubernetes.io/projected/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-kube-api-access-t4rp5\") pod \"maas-api-revoke-sub-a948a352-ljt7l\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.270962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.270939 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:06.348482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.348449 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:06.547019 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:06.546995 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l"] Nov 04 00:44:06.549540 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:44:06.549511 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67a52df_2c6c_4eba_877f_e9a87e3d18e1.slice/crio-225db6b236e1230d1f61cd64d32082b750dc382010198f7102b93f5fd9e8cd32 WatchSource:0}: Error finding container 225db6b236e1230d1f61cd64d32082b750dc382010198f7102b93f5fd9e8cd32: Status 404 returned error can't find the container with id 225db6b236e1230d1f61cd64d32082b750dc382010198f7102b93f5fd9e8cd32 Nov 04 00:44:07.160077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:07.159973 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:07.381203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:07.381153 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" event={"ID":"a67a52df-2c6c-4eba-877f-e9a87e3d18e1","Type":"ContainerStarted","Data":"6061c3ef475b9c16d8e5a80e597b9c4fdd33a91b60584446f8e10eae236380f7"} Nov 04 00:44:07.381404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:07.381219 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" event={"ID":"a67a52df-2c6c-4eba-877f-e9a87e3d18e1","Type":"ContainerStarted","Data":"225db6b236e1230d1f61cd64d32082b750dc382010198f7102b93f5fd9e8cd32"} Nov 04 00:44:07.405371 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:07.405318 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" podStartSLOduration=2.40530516 podStartE2EDuration="2.40530516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:44:07.402227556 +0000 UTC m=+4466.224823721" watchObservedRunningTime="2026-11-04 00:44:07.40530516 +0000 UTC m=+4466.227901364" Nov 04 00:44:07.935493 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:07.935452 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:08.388428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:08.388392 6208 generic.go:356] "Generic (PLEG): container finished" podID="a67a52df-2c6c-4eba-877f-e9a87e3d18e1" containerID="6061c3ef475b9c16d8e5a80e597b9c4fdd33a91b60584446f8e10eae236380f7" exitCode=0 Nov 04 00:44:08.388842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:08.388478 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" event={"ID":"a67a52df-2c6c-4eba-877f-e9a87e3d18e1","Type":"ContainerDied","Data":"6061c3ef475b9c16d8e5a80e597b9c4fdd33a91b60584446f8e10eae236380f7"} Nov 04 00:44:08.745365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:08.745313 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:09.529818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.529791 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:09.560365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.560336 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-kube-api-access-t4rp5\" (UniqueName: \"kubernetes.io/projected/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-kube-api-access-t4rp5\") pod \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " Nov 04 00:44:09.560537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.560406 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-openshift-ca\") pod \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " Nov 04 00:44:09.560537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.560473 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-xks-ca\") pod \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\" (UID: \"a67a52df-2c6c-4eba-877f-e9a87e3d18e1\") " Nov 04 00:44:09.560802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.560778 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-openshift-ca" pod "a67a52df-2c6c-4eba-877f-e9a87e3d18e1" (UID: "a67a52df-2c6c-4eba-877f-e9a87e3d18e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:44:09.562518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.562496 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-kube-api-access-t4rp5" pod "a67a52df-2c6c-4eba-877f-e9a87e3d18e1" (UID: "a67a52df-2c6c-4eba-877f-e9a87e3d18e1"). InnerVolumeSpecName "kube-api-access-t4rp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:44:09.562518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.562506 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-xks-ca" pod "a67a52df-2c6c-4eba-877f-e9a87e3d18e1" (UID: "a67a52df-2c6c-4eba-877f-e9a87e3d18e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:44:09.654298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.654212 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:09.661459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.661434 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4rp5\" (UniqueName: \"kubernetes.io/projected/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-kube-api-access-t4rp5\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:44:09.661459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.661461 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:44:09.661601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:09.661474 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67a52df-2c6c-4eba-877f-e9a87e3d18e1-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:44:10.394209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:10.394166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:10.399459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:10.399434 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" Nov 04 00:44:10.399595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:10.399474 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l" event={"ID":"a67a52df-2c6c-4eba-877f-e9a87e3d18e1","Type":"ContainerDied","Data":"225db6b236e1230d1f61cd64d32082b750dc382010198f7102b93f5fd9e8cd32"} Nov 04 00:44:10.399595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:10.399520 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="225db6b236e1230d1f61cd64d32082b750dc382010198f7102b93f5fd9e8cd32" Nov 04 00:44:11.144894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:11.144858 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:11.939444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:11.939406 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:12.734175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:12.734129 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:13.533343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:13.533311 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:14.359650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:14.359619 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:15.128165 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:15.128129 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:15.834972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:15.834937 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:16.704370 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:16.704333 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:16.801658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:16.801623 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:17.197915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:17.197871 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:17.535662 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:17.535616 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:18.244198 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:18.244162 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:19.034899 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:19.034859 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:19.751023 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:19.750985 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:20.516265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:20.516229 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:21.232551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:21.232513 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:21.664657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:21.664575 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm"] Nov 04 00:44:21.759976 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:21.759941 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:22.025108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:22.025069 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:22.743376 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:22.743334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:22.913659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:22.913618 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:23.013541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:23.013455 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:23.540026 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:23.539986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:24.186588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:24.186548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:24.296419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:24.296386 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:24.396491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:24.396453 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:30.570330 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:30.570290 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:30.670259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:30.670221 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:35.706435 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:44:35.706394 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:44:36.027728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:36.027631 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf"] Nov 04 00:44:36.029598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:36.029572 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xxnnf"] Nov 04 00:44:36.684833 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:36.684793 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" podUID="25b1256b-1b2e-4c1a-9134-1ced4b096414" containerName="main" containerID="cri-o://a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0" gracePeriod=45 Nov 04 00:44:36.944564 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:36.944501 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:44:37.020672 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.020639 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b1256b-1b2e-4c1a-9134-1ced4b096414-tls-certs\" (UniqueName: \"kubernetes.io/secret/25b1256b-1b2e-4c1a-9134-1ced4b096414-tls-certs\") pod \"25b1256b-1b2e-4c1a-9134-1ced4b096414\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " Nov 04 00:44:37.020864 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.020687 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-model-cache\") pod \"25b1256b-1b2e-4c1a-9134-1ced4b096414\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " Nov 04 00:44:37.020864 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.020712 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b1256b-1b2e-4c1a-9134-1ced4b096414-kube-api-access-65qsf\" (UniqueName: \"kubernetes.io/projected/25b1256b-1b2e-4c1a-9134-1ced4b096414-kube-api-access-65qsf\") pod \"25b1256b-1b2e-4c1a-9134-1ced4b096414\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " Nov 04 00:44:37.020864 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.020735 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-tmp-dir\") pod \"25b1256b-1b2e-4c1a-9134-1ced4b096414\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " Nov 04 00:44:37.020864 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.020776 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-home\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-home\") pod \"25b1256b-1b2e-4c1a-9134-1ced4b096414\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " Nov 04 00:44:37.021161 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.020950 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-dshm\") pod \"25b1256b-1b2e-4c1a-9134-1ced4b096414\" (UID: \"25b1256b-1b2e-4c1a-9134-1ced4b096414\") " Nov 04 00:44:37.021161 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.021015 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-model-cache" pod "25b1256b-1b2e-4c1a-9134-1ced4b096414" (UID: "25b1256b-1b2e-4c1a-9134-1ced4b096414"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:44:37.021273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.021112 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-tmp-dir" pod "25b1256b-1b2e-4c1a-9134-1ced4b096414" (UID: "25b1256b-1b2e-4c1a-9134-1ced4b096414"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:44:37.021273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.021177 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-home" pod "25b1256b-1b2e-4c1a-9134-1ced4b096414" (UID: "25b1256b-1b2e-4c1a-9134-1ced4b096414"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:44:37.021383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.021330 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:44:37.021383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.021349 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:44:37.021383 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.021364 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:44:37.023272 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.023248 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b1256b-1b2e-4c1a-9134-1ced4b096414-kube-api-access-65qsf" pod "25b1256b-1b2e-4c1a-9134-1ced4b096414" (UID: "25b1256b-1b2e-4c1a-9134-1ced4b096414"). InnerVolumeSpecName "kube-api-access-65qsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:44:37.023494 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.023477 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b1256b-1b2e-4c1a-9134-1ced4b096414-tls-certs" pod "25b1256b-1b2e-4c1a-9134-1ced4b096414" (UID: "25b1256b-1b2e-4c1a-9134-1ced4b096414"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:44:37.023567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.023488 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-dshm" pod "25b1256b-1b2e-4c1a-9134-1ced4b096414" (UID: "25b1256b-1b2e-4c1a-9134-1ced4b096414"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:44:37.122363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.122323 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b1256b-1b2e-4c1a-9134-1ced4b096414-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:44:37.122363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.122361 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b1256b-1b2e-4c1a-9134-1ced4b096414-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:44:37.122588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.122374 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qsf\" (UniqueName: \"kubernetes.io/projected/25b1256b-1b2e-4c1a-9134-1ced4b096414-kube-api-access-65qsf\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:44:37.293300 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.293268 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db508a9-460f-4db6-a334-5ed4e302ee0a" path="/var/lib/kubelet/pods/2db508a9-460f-4db6-a334-5ed4e302ee0a/volumes" Nov 04 00:44:37.549100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.548990 6208 generic.go:356] "Generic (PLEG): container finished" podID="25b1256b-1b2e-4c1a-9134-1ced4b096414" containerID="a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0" exitCode=0 Nov 04 00:44:37.549100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.549070 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" Nov 04 00:44:37.549100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.549080 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" event={"ID":"25b1256b-1b2e-4c1a-9134-1ced4b096414","Type":"ContainerDied","Data":"a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0"} Nov 04 00:44:37.549399 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.549124 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm" event={"ID":"25b1256b-1b2e-4c1a-9134-1ced4b096414","Type":"ContainerDied","Data":"e3c6b090307414dfec01b87f5938a8cee5bee60b5c2b0c8cfdcf79b51f5201dc"} Nov 04 00:44:37.549399 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.549141 6208 scope.go:122] "RemoveContainer" containerID="a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0" Nov 04 00:44:37.558088 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.558071 6208 scope.go:122] "RemoveContainer" containerID="a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0" Nov 04 00:44:37.558386 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:44:37.558363 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0\": container with ID starting with a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0 not found: ID does not exist" containerID="a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0" Nov 04 00:44:37.558460 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.558398 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0"} err="failed to get container status \"a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0\": rpc error: code = NotFound desc = could not find container \"a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0\": container with ID starting with a74e055116577f783872af9db89d619ffa3c56dcc5994a36d91aeef3237172e0 not found: ID does not exist" Nov 04 00:44:37.567259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.567238 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm"] Nov 04 00:44:37.571075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:37.571029 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ca4fbe/simulator-single-main-ca4fbe-kserve-7bc6759bdf-8wdnm"] Nov 04 00:44:39.293956 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:39.293916 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b1256b-1b2e-4c1a-9134-1ced4b096414" path="/var/lib/kubelet/pods/25b1256b-1b2e-4c1a-9134-1ced4b096414/volumes" Nov 04 00:44:43.108016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:43.107984 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:44:43.113185 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:43.113161 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:44:46.307883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:46.307847 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:46.403119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:46.403072 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:54.585186 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:54.585110 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:54.684972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:54.684936 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:55.773426 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.773391 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:55.799133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.799103 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8"] Nov 04 00:44:55.799479 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.799467 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a67a52df-2c6c-4eba-877f-e9a87e3d18e1" containerName="revoke-keys" Nov 04 00:44:55.799530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.799480 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67a52df-2c6c-4eba-877f-e9a87e3d18e1" containerName="revoke-keys" Nov 04 00:44:55.799530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.799500 6208 cpu_manager.go:414] "containerMap: removing container" podUID="25b1256b-1b2e-4c1a-9134-1ced4b096414" containerName="main" Nov 04 00:44:55.799530 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.799505 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b1256b-1b2e-4c1a-9134-1ced4b096414" containerName="main" Nov 04 00:44:55.799628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.799565 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b1256b-1b2e-4c1a-9134-1ced4b096414" containerName="main" Nov 04 00:44:55.799628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.799579 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67a52df-2c6c-4eba-877f-e9a87e3d18e1" containerName="revoke-keys" Nov 04 00:44:55.802824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.802802 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:44:55.812093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.812026 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs"] Nov 04 00:44:55.816925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.816895 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ca4fbe\"" Nov 04 00:44:55.819675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.817718 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:55.823478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.823454 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8"] Nov 04 00:44:55.830060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.830021 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xflt\"" Nov 04 00:44:55.830846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.830136 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ca4fbe-serving-cert\"" Nov 04 00:44:55.830846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.830176 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 00:44:55.832251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.832227 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs"] Nov 04 00:44:55.841879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.841856 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6"] Nov 04 00:44:55.845830 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.845809 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:44:55.856243 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.856205 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6"] Nov 04 00:44:55.884296 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.884259 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:55.905268 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.905238 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmb75\" (UniqueName: \"kubernetes.io/projected/2725250c-844d-4435-8070-7ddfc9841238-kube-api-access-zmb75\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:55.905482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.905308 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2725250c-844d-4435-8070-7ddfc9841238-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:55.905482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.905373 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5af56c-4637-45cd-a84d-b2e5896b082c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6\" (UID: \"4e5af56c-4637-45cd-a84d-b2e5896b082c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:44:55.905482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.905414 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:55.905482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.905447 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22aa9d0-11be-4971-b00e-09405d355e86-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8\" (UID: \"f22aa9d0-11be-4971-b00e-09405d355e86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:44:55.905935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.905580 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:55.905935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.905617 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4rk\" (UniqueName: \"kubernetes.io/projected/4e5af56c-4637-45cd-a84d-b2e5896b082c-kube-api-access-qx4rk\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6\" (UID: \"4e5af56c-4637-45cd-a84d-b2e5896b082c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:44:55.905935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.905710 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5ljq\" (UniqueName: \"kubernetes.io/projected/f22aa9d0-11be-4971-b00e-09405d355e86-kube-api-access-d5ljq\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8\" (UID: \"f22aa9d0-11be-4971-b00e-09405d355e86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:44:55.916761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.916734 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:44:55.916761 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.916753 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:44:55.925954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.925928 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0b757b52-dockercfg-rzcq4\"" Nov 04 00:44:55.979988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:55.979951 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:56.006289 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.006256 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4rk\" (UniqueName: \"kubernetes.io/projected/4e5af56c-4637-45cd-a84d-b2e5896b082c-kube-api-access-qx4rk\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6\" (UID: \"4e5af56c-4637-45cd-a84d-b2e5896b082c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:44:56.006478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.006331 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5ljq\" (UniqueName: \"kubernetes.io/projected/f22aa9d0-11be-4971-b00e-09405d355e86-kube-api-access-d5ljq\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8\" (UID: \"f22aa9d0-11be-4971-b00e-09405d355e86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:44:56.006478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.006378 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmb75\" (UniqueName: \"kubernetes.io/projected/2725250c-844d-4435-8070-7ddfc9841238-kube-api-access-zmb75\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:56.006478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.006436 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2725250c-844d-4435-8070-7ddfc9841238-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:56.006478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.006468 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5af56c-4637-45cd-a84d-b2e5896b082c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6\" (UID: \"4e5af56c-4637-45cd-a84d-b2e5896b082c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:44:56.006695 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.006512 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:56.006695 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.006555 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22aa9d0-11be-4971-b00e-09405d355e86-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8\" (UID: \"f22aa9d0-11be-4971-b00e-09405d355e86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:44:56.006695 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.006618 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:56.007431 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.007406 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22aa9d0-11be-4971-b00e-09405d355e86-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8\" (UID: \"f22aa9d0-11be-4971-b00e-09405d355e86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:44:56.007536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.007491 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5af56c-4637-45cd-a84d-b2e5896b082c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6\" (UID: \"4e5af56c-4637-45cd-a84d-b2e5896b082c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:44:56.007689 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.007672 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2725250c-844d-4435-8070-7ddfc9841238-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:56.009078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.009026 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:56.009167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.009121 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:56.018936 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.018914 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4rk\" (UniqueName: \"kubernetes.io/projected/4e5af56c-4637-45cd-a84d-b2e5896b082c-kube-api-access-qx4rk\") pod \"payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6\" (UID: \"4e5af56c-4637-45cd-a84d-b2e5896b082c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:44:56.019221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.019202 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmb75\" (UniqueName: \"kubernetes.io/projected/2725250c-844d-4435-8070-7ddfc9841238-kube-api-access-zmb75\") pod \"maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:56.019295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.019231 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5ljq\" (UniqueName: \"kubernetes.io/projected/f22aa9d0-11be-4971-b00e-09405d355e86-kube-api-access-d5ljq\") pod \"payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8\" (UID: \"f22aa9d0-11be-4971-b00e-09405d355e86\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:44:56.084759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.084671 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:56.134558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.134525 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:44:56.149073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.149013 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:56.160577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.160526 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:44:56.417400 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.417374 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8"] Nov 04 00:44:56.419590 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:44:56.419562 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22aa9d0_11be_4971_b00e_09405d355e86.slice/crio-f0bb83be31cfa26a80cf5f55a81dd873e87e60183e6d14685e1a560cc8f3b8ca WatchSource:0}: Error finding container f0bb83be31cfa26a80cf5f55a81dd873e87e60183e6d14685e1a560cc8f3b8ca: Status 404 returned error can't find the container with id f0bb83be31cfa26a80cf5f55a81dd873e87e60183e6d14685e1a560cc8f3b8ca Nov 04 00:44:56.455709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.455680 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6"] Nov 04 00:44:56.456800 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:44:56.456775 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5af56c_4637_45cd_a84d_b2e5896b082c.slice/crio-5346628160f55f2cd1745ef29d194bb3613e1968b3fd9aa289dbcf12d206f216 WatchSource:0}: Error finding container 5346628160f55f2cd1745ef29d194bb3613e1968b3fd9aa289dbcf12d206f216: Status 404 returned error can't find the container with id 5346628160f55f2cd1745ef29d194bb3613e1968b3fd9aa289dbcf12d206f216 Nov 04 00:44:56.458357 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.458333 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs"] Nov 04 00:44:56.459866 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:44:56.459837 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2725250c_844d_4435_8070_7ddfc9841238.slice/crio-0f9b69a065a235afffa3aabe40b9d43e7b13fa2a392c25a5371fc8086fcb88a1 WatchSource:0}: Error finding container 0f9b69a065a235afffa3aabe40b9d43e7b13fa2a392c25a5371fc8086fcb88a1: Status 404 returned error can't find the container with id 0f9b69a065a235afffa3aabe40b9d43e7b13fa2a392c25a5371fc8086fcb88a1 Nov 04 00:44:56.650527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.650484 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" event={"ID":"f22aa9d0-11be-4971-b00e-09405d355e86","Type":"ContainerStarted","Data":"5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7"} Nov 04 00:44:56.650745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.650548 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" event={"ID":"f22aa9d0-11be-4971-b00e-09405d355e86","Type":"ContainerStarted","Data":"f0bb83be31cfa26a80cf5f55a81dd873e87e60183e6d14685e1a560cc8f3b8ca"} Nov 04 00:44:56.650745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.650653 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:44:56.652111 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.652083 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" event={"ID":"4e5af56c-4637-45cd-a84d-b2e5896b082c","Type":"ContainerStarted","Data":"43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad"} Nov 04 00:44:56.652266 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.652147 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" event={"ID":"4e5af56c-4637-45cd-a84d-b2e5896b082c","Type":"ContainerStarted","Data":"5346628160f55f2cd1745ef29d194bb3613e1968b3fd9aa289dbcf12d206f216"} Nov 04 00:44:56.652266 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.652178 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:44:56.653360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.653327 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" event={"ID":"2725250c-844d-4435-8070-7ddfc9841238","Type":"ContainerStarted","Data":"0f9b69a065a235afffa3aabe40b9d43e7b13fa2a392c25a5371fc8086fcb88a1"} Nov 04 00:44:56.671182 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.671075 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" podStartSLOduration=1.671059981 podStartE2EDuration="1.671059981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:44:56.669116681 +0000 UTC m=+4515.491712846" watchObservedRunningTime="2026-11-04 00:44:56.671059981 +0000 UTC m=+4515.493656142" Nov 04 00:44:56.692516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.692463 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" podStartSLOduration=1.692449791 podStartE2EDuration="1.692449791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:44:56.690682061 +0000 UTC m=+4515.513278224" watchObservedRunningTime="2026-11-04 00:44:56.692449791 +0000 UTC m=+4515.515045947" Nov 04 00:44:56.880368 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:56.880334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:57.637999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:57.637914 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:57.659874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:57.659837 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" event={"ID":"2725250c-844d-4435-8070-7ddfc9841238","Type":"ContainerStarted","Data":"38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc"} Nov 04 00:44:57.660064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:57.659921 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:44:57.681334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:57.681286 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" podStartSLOduration=1.8695400819999999 podStartE2EDuration="2.681271468s" totalImagesPullingTime="811.731386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:44:57.680377717 +0000 UTC m=+4516.502973879" watchObservedRunningTime="2026-11-04 00:44:57.681271468 +0000 UTC m=+4516.503867655" Nov 04 00:44:57.738092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:57.738059 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:58.443912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:58.443879 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:58.538565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:58.538534 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:59.245460 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:59.245422 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:59.368580 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:59.368531 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:44:59.593716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:44:59.593623 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:00.039903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.039867 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:00.148599 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.148563 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n"] Nov 04 00:45:00.153667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.153632 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:00.165699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.165573 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n"] Nov 04 00:45:00.246766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.246716 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1162c916-4c68-4361-b7b2-963902ab586a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895885-xdb6n\" (UID: \"1162c916-4c68-4361-b7b2-963902ab586a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:00.246963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.246848 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8r6c\" (UniqueName: \"kubernetes.io/projected/1162c916-4c68-4361-b7b2-963902ab586a-kube-api-access-h8r6c\") pod \"maas-api-key-cleanup-29895885-xdb6n\" (UID: \"1162c916-4c68-4361-b7b2-963902ab586a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:00.255085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.255060 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:45:00.347558 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.347458 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1162c916-4c68-4361-b7b2-963902ab586a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895885-xdb6n\" (UID: \"1162c916-4c68-4361-b7b2-963902ab586a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:00.347755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.347723 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8r6c\" (UniqueName: \"kubernetes.io/projected/1162c916-4c68-4361-b7b2-963902ab586a-kube-api-access-h8r6c\") pod \"maas-api-key-cleanup-29895885-xdb6n\" (UID: \"1162c916-4c68-4361-b7b2-963902ab586a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:00.348155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.348133 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1162c916-4c68-4361-b7b2-963902ab586a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895885-xdb6n\" (UID: \"1162c916-4c68-4361-b7b2-963902ab586a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:00.363877 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.363845 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8r6c\" (UniqueName: \"kubernetes.io/projected/1162c916-4c68-4361-b7b2-963902ab586a-kube-api-access-h8r6c\") pod \"maas-api-key-cleanup-29895885-xdb6n\" (UID: \"1162c916-4c68-4361-b7b2-963902ab586a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:00.474303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.474266 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:00.753750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.753721 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n"] Nov 04 00:45:00.755641 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:45:00.755607 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1162c916_4c68_4361_b7b2_963902ab586a.slice/crio-e87384394da607881a8c54f73ec482df1ac9e835d69cec9e7d6f72d33884b0c4 WatchSource:0}: Error finding container e87384394da607881a8c54f73ec482df1ac9e835d69cec9e7d6f72d33884b0c4: Status 404 returned error can't find the container with id e87384394da607881a8c54f73ec482df1ac9e835d69cec9e7d6f72d33884b0c4 Nov 04 00:45:00.845932 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:00.845896 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:01.051171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:01.051076 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:01.643565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:01.643523 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:01.685210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:01.685103 6208 generic.go:356] "Generic (PLEG): container finished" podID="1162c916-4c68-4361-b7b2-963902ab586a" containerID="b947f582ee0d7f5e2b989b9ff661a685932b39b67f5e298e42425eb59ccb532b" exitCode=60 Nov 04 00:45:01.685210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:01.685146 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" event={"ID":"1162c916-4c68-4361-b7b2-963902ab586a","Type":"ContainerDied","Data":"b947f582ee0d7f5e2b989b9ff661a685932b39b67f5e298e42425eb59ccb532b"} Nov 04 00:45:01.685979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:01.685192 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" event={"ID":"1162c916-4c68-4361-b7b2-963902ab586a","Type":"ContainerStarted","Data":"e87384394da607881a8c54f73ec482df1ac9e835d69cec9e7d6f72d33884b0c4"} Nov 04 00:45:01.686406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:01.686385 6208 scope.go:122] "RemoveContainer" containerID="b947f582ee0d7f5e2b989b9ff661a685932b39b67f5e298e42425eb59ccb532b" Nov 04 00:45:02.446345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:02.446262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:02.693241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:02.693204 6208 generic.go:356] "Generic (PLEG): container finished" podID="1162c916-4c68-4361-b7b2-963902ab586a" containerID="fc7ac29be1e30182a83f1249454c20798404074fbd97d8b872b755ac12d98c85" exitCode=60 Nov 04 00:45:02.693440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:02.693284 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" event={"ID":"1162c916-4c68-4361-b7b2-963902ab586a","Type":"ContainerDied","Data":"fc7ac29be1e30182a83f1249454c20798404074fbd97d8b872b755ac12d98c85"} Nov 04 00:45:02.693440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:02.693352 6208 scope.go:122] "RemoveContainer" containerID="b947f582ee0d7f5e2b989b9ff661a685932b39b67f5e298e42425eb59ccb532b" Nov 04 00:45:02.693573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:02.693542 6208 scope.go:122] "RemoveContainer" containerID="fc7ac29be1e30182a83f1249454c20798404074fbd97d8b872b755ac12d98c85" Nov 04 00:45:02.693817 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:45:02.693795 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895885-xdb6n_odh-ai-gateway-infra(1162c916-4c68-4361-b7b2-963902ab586a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" podUID="1162c916-4c68-4361-b7b2-963902ab586a" Nov 04 00:45:03.156863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:03.156760 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:03.670390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:03.670361 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:45:03.702429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:03.702396 6208 scope.go:122] "RemoveContainer" containerID="fc7ac29be1e30182a83f1249454c20798404074fbd97d8b872b755ac12d98c85" Nov 04 00:45:03.702920 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:45:03.702702 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895885-xdb6n_odh-ai-gateway-infra(1162c916-4c68-4361-b7b2-963902ab586a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" podUID="1162c916-4c68-4361-b7b2-963902ab586a" Nov 04 00:45:04.042795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:04.042754 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:04.872446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:04.872398 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:05.002738 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:05.002703 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:05.705689 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:45:05.705642 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:45:05.718317 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:05.718286 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:06.442653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:06.442603 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:07.157094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:07.157061 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:07.663021 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:07.662989 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:45:07.663461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:07.663079 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:45:07.934425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:07.934337 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:08.732581 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:08.732545 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:08.850332 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:08.850299 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:08.951356 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:08.951309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:09.513259 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:09.513216 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:10.247824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:10.247789 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:11.029446 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:11.029411 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:11.779104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:11.779064 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:12.631590 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:12.631554 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:13.287907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:13.287875 6208 scope.go:122] "RemoveContainer" containerID="fc7ac29be1e30182a83f1249454c20798404074fbd97d8b872b755ac12d98c85" Nov 04 00:45:13.369218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:13.369182 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:13.762011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:13.761974 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" event={"ID":"1162c916-4c68-4361-b7b2-963902ab586a","Type":"ContainerStarted","Data":"4ed2fe1186f42df1566b6f9837040f057546a137a04fe5128ce143db297f45c7"} Nov 04 00:45:13.780164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:13.780117 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" podStartSLOduration=13.780106302 podStartE2EDuration="13.780106302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:45:13.779802982 +0000 UTC m=+4532.602399146" watchObservedRunningTime="2026-11-04 00:45:13.780106302 +0000 UTC m=+4532.602702464" Nov 04 00:45:14.136879 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.136844 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:14.709171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.709136 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs"] Nov 04 00:45:14.709828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.709798 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" podUID="2725250c-844d-4435-8070-7ddfc9841238" containerName="maas-api" containerID="cri-o://38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc" gracePeriod=30 Nov 04 00:45:14.759149 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.759116 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:14.769917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.769886 6208 generic.go:356] "Generic (PLEG): container finished" podID="1162c916-4c68-4361-b7b2-963902ab586a" containerID="4ed2fe1186f42df1566b6f9837040f057546a137a04fe5128ce143db297f45c7" exitCode=60 Nov 04 00:45:14.770084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.769933 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" event={"ID":"1162c916-4c68-4361-b7b2-963902ab586a","Type":"ContainerDied","Data":"4ed2fe1186f42df1566b6f9837040f057546a137a04fe5128ce143db297f45c7"} Nov 04 00:45:14.770084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.769987 6208 scope.go:122] "RemoveContainer" containerID="fc7ac29be1e30182a83f1249454c20798404074fbd97d8b872b755ac12d98c85" Nov 04 00:45:14.770367 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.770349 6208 scope.go:122] "RemoveContainer" containerID="4ed2fe1186f42df1566b6f9837040f057546a137a04fe5128ce143db297f45c7" Nov 04 00:45:14.770642 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:45:14.770623 6208 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895885-xdb6n_odh-ai-gateway-infra(1162c916-4c68-4361-b7b2-963902ab586a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" podUID="1162c916-4c68-4361-b7b2-963902ab586a" Nov 04 00:45:14.789521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.789495 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8"] Nov 04 00:45:14.796971 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.796946 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6"] Nov 04 00:45:14.800086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.800057 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n"] Nov 04 00:45:14.881510 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.881457 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:14.973426 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.973256 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:45:14.982405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:14.982373 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:15.093587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.093555 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-metrics-certs\") pod \"2725250c-844d-4435-8070-7ddfc9841238\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " Nov 04 00:45:15.093792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.093618 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-maas-api-tls\") pod \"2725250c-844d-4435-8070-7ddfc9841238\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " Nov 04 00:45:15.093792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.093694 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2725250c-844d-4435-8070-7ddfc9841238-kube-api-access-zmb75\" (UniqueName: \"kubernetes.io/projected/2725250c-844d-4435-8070-7ddfc9841238-kube-api-access-zmb75\") pod \"2725250c-844d-4435-8070-7ddfc9841238\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " Nov 04 00:45:15.093792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.093771 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2725250c-844d-4435-8070-7ddfc9841238-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2725250c-844d-4435-8070-7ddfc9841238-openshift-service-ca\") pod \"2725250c-844d-4435-8070-7ddfc9841238\" (UID: \"2725250c-844d-4435-8070-7ddfc9841238\") " Nov 04 00:45:15.094208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.094181 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2725250c-844d-4435-8070-7ddfc9841238-openshift-service-ca" pod "2725250c-844d-4435-8070-7ddfc9841238" (UID: "2725250c-844d-4435-8070-7ddfc9841238"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:45:15.095804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.095773 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-metrics-certs" pod "2725250c-844d-4435-8070-7ddfc9841238" (UID: "2725250c-844d-4435-8070-7ddfc9841238"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:45:15.095804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.095796 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-maas-api-tls" pod "2725250c-844d-4435-8070-7ddfc9841238" (UID: "2725250c-844d-4435-8070-7ddfc9841238"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:45:15.095938 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.095820 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2725250c-844d-4435-8070-7ddfc9841238-kube-api-access-zmb75" pod "2725250c-844d-4435-8070-7ddfc9841238" (UID: "2725250c-844d-4435-8070-7ddfc9841238"). InnerVolumeSpecName "kube-api-access-zmb75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:45:15.112927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.112900 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:15.195255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.195216 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2725250c-844d-4435-8070-7ddfc9841238-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:15.195255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.195258 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:15.195493 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.195273 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2725250c-844d-4435-8070-7ddfc9841238-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:15.195493 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.195287 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmb75\" (UniqueName: \"kubernetes.io/projected/2725250c-844d-4435-8070-7ddfc9841238-kube-api-access-zmb75\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:15.206210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.206178 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:15.566026 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.565946 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:15.776354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.776322 6208 generic.go:356] "Generic (PLEG): container finished" podID="2725250c-844d-4435-8070-7ddfc9841238" containerID="38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc" exitCode=0 Nov 04 00:45:15.776951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.776367 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" Nov 04 00:45:15.776951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.776397 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" event={"ID":"2725250c-844d-4435-8070-7ddfc9841238","Type":"ContainerDied","Data":"38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc"} Nov 04 00:45:15.776951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.776444 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs" event={"ID":"2725250c-844d-4435-8070-7ddfc9841238","Type":"ContainerDied","Data":"0f9b69a065a235afffa3aabe40b9d43e7b13fa2a392c25a5371fc8086fcb88a1"} Nov 04 00:45:15.776951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.776461 6208 scope.go:122] "RemoveContainer" containerID="38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc" Nov 04 00:45:15.786178 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.786157 6208 scope.go:122] "RemoveContainer" containerID="38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc" Nov 04 00:45:15.786437 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:45:15.786420 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc\": container with ID starting with 38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc not found: ID does not exist" containerID="38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc" Nov 04 00:45:15.786499 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.786445 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc"} err="failed to get container status \"38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc\": rpc error: code = NotFound desc = could not find container \"38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc\": container with ID starting with 38fa94aa2e3408e68e51ca3a5254c23a2bbeef893dd012424c9b4033cd5bebdc not found: ID does not exist" Nov 04 00:45:15.796397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.796370 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs"] Nov 04 00:45:15.798975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.798954 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ca4fbe-655f9c7d94-sjpzs"] Nov 04 00:45:15.917908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:15.917878 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:16.002351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.002310 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1162c916-4c68-4361-b7b2-963902ab586a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1162c916-4c68-4361-b7b2-963902ab586a-maas-api-service-ca\") pod \"1162c916-4c68-4361-b7b2-963902ab586a\" (UID: \"1162c916-4c68-4361-b7b2-963902ab586a\") " Nov 04 00:45:16.002521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.002469 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1162c916-4c68-4361-b7b2-963902ab586a-kube-api-access-h8r6c\" (UniqueName: \"kubernetes.io/projected/1162c916-4c68-4361-b7b2-963902ab586a-kube-api-access-h8r6c\") pod \"1162c916-4c68-4361-b7b2-963902ab586a\" (UID: \"1162c916-4c68-4361-b7b2-963902ab586a\") " Nov 04 00:45:16.002667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.002643 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1162c916-4c68-4361-b7b2-963902ab586a-maas-api-service-ca" pod "1162c916-4c68-4361-b7b2-963902ab586a" (UID: "1162c916-4c68-4361-b7b2-963902ab586a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:45:16.002781 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.002767 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1162c916-4c68-4361-b7b2-963902ab586a-maas-api-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:16.004451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.004426 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1162c916-4c68-4361-b7b2-963902ab586a-kube-api-access-h8r6c" pod "1162c916-4c68-4361-b7b2-963902ab586a" (UID: "1162c916-4c68-4361-b7b2-963902ab586a"). InnerVolumeSpecName "kube-api-access-h8r6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:45:16.111565 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.104759 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8r6c\" (UniqueName: \"kubernetes.io/projected/1162c916-4c68-4361-b7b2-963902ab586a-kube-api-access-h8r6c\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:16.786326 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.786291 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" Nov 04 00:45:16.786807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.786324 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n" event={"ID":"1162c916-4c68-4361-b7b2-963902ab586a","Type":"ContainerDied","Data":"e87384394da607881a8c54f73ec482df1ac9e835d69cec9e7d6f72d33884b0c4"} Nov 04 00:45:16.786807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.786384 6208 scope.go:122] "RemoveContainer" containerID="4ed2fe1186f42df1566b6f9837040f057546a137a04fe5128ce143db297f45c7" Nov 04 00:45:16.810143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.810110 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n"] Nov 04 00:45:16.814858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:16.814835 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895885-xdb6n"] Nov 04 00:45:17.032314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:17.032279 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw"] Nov 04 00:45:17.037716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:17.037640 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pjzhw"] Nov 04 00:45:17.294650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:17.294571 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1162c916-4c68-4361-b7b2-963902ab586a" path="/var/lib/kubelet/pods/1162c916-4c68-4361-b7b2-963902ab586a/volumes" Nov 04 00:45:17.295012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:17.294998 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2725250c-844d-4435-8070-7ddfc9841238" path="/var/lib/kubelet/pods/2725250c-844d-4435-8070-7ddfc9841238/volumes" Nov 04 00:45:17.295403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:17.295389 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2112d05-ca0d-4e9b-819e-66767cf9c2e4" path="/var/lib/kubelet/pods/e2112d05-ca0d-4e9b-819e-66767cf9c2e4/volumes" Nov 04 00:45:19.790051 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:19.790000 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" podUID="f22aa9d0-11be-4971-b00e-09405d355e86" containerName="payload-processing" containerID="cri-o://5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7" gracePeriod=25 Nov 04 00:45:19.797501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:19.797470 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" podUID="4e5af56c-4637-45cd-a84d-b2e5896b082c" containerName="payload-pre-processing" containerID="cri-o://43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad" gracePeriod=25 Nov 04 00:45:20.036387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.036354 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:45:20.082915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.082887 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:45:20.146893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.146857 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22aa9d0-11be-4971-b00e-09405d355e86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22aa9d0-11be-4971-b00e-09405d355e86-plugins-config-volume\") pod \"f22aa9d0-11be-4971-b00e-09405d355e86\" (UID: \"f22aa9d0-11be-4971-b00e-09405d355e86\") " Nov 04 00:45:20.147137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.146929 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22aa9d0-11be-4971-b00e-09405d355e86-kube-api-access-d5ljq\" (UniqueName: \"kubernetes.io/projected/f22aa9d0-11be-4971-b00e-09405d355e86-kube-api-access-d5ljq\") pod \"f22aa9d0-11be-4971-b00e-09405d355e86\" (UID: \"f22aa9d0-11be-4971-b00e-09405d355e86\") " Nov 04 00:45:20.147345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.147321 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22aa9d0-11be-4971-b00e-09405d355e86-plugins-config-volume" pod "f22aa9d0-11be-4971-b00e-09405d355e86" (UID: "f22aa9d0-11be-4971-b00e-09405d355e86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:45:20.149005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.148983 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22aa9d0-11be-4971-b00e-09405d355e86-kube-api-access-d5ljq" pod "f22aa9d0-11be-4971-b00e-09405d355e86" (UID: "f22aa9d0-11be-4971-b00e-09405d355e86"). InnerVolumeSpecName "kube-api-access-d5ljq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:45:20.247822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.247779 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5af56c-4637-45cd-a84d-b2e5896b082c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5af56c-4637-45cd-a84d-b2e5896b082c-plugins-config-volume\") pod \"4e5af56c-4637-45cd-a84d-b2e5896b082c\" (UID: \"4e5af56c-4637-45cd-a84d-b2e5896b082c\") " Nov 04 00:45:20.248060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.247891 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5af56c-4637-45cd-a84d-b2e5896b082c-kube-api-access-qx4rk\" (UniqueName: \"kubernetes.io/projected/4e5af56c-4637-45cd-a84d-b2e5896b082c-kube-api-access-qx4rk\") pod \"4e5af56c-4637-45cd-a84d-b2e5896b082c\" (UID: \"4e5af56c-4637-45cd-a84d-b2e5896b082c\") " Nov 04 00:45:20.248154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.248118 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f22aa9d0-11be-4971-b00e-09405d355e86-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:20.248154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.248135 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5ljq\" (UniqueName: \"kubernetes.io/projected/f22aa9d0-11be-4971-b00e-09405d355e86-kube-api-access-d5ljq\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:20.248307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.248279 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5af56c-4637-45cd-a84d-b2e5896b082c-plugins-config-volume" pod "4e5af56c-4637-45cd-a84d-b2e5896b082c" (UID: "4e5af56c-4637-45cd-a84d-b2e5896b082c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:45:20.249940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.249920 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5af56c-4637-45cd-a84d-b2e5896b082c-kube-api-access-qx4rk" pod "4e5af56c-4637-45cd-a84d-b2e5896b082c" (UID: "4e5af56c-4637-45cd-a84d-b2e5896b082c"). InnerVolumeSpecName "kube-api-access-qx4rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:45:20.349262 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.349166 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5af56c-4637-45cd-a84d-b2e5896b082c-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:20.349262 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.349210 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx4rk\" (UniqueName: \"kubernetes.io/projected/4e5af56c-4637-45cd-a84d-b2e5896b082c-kube-api-access-qx4rk\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:20.813941 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.813908 6208 generic.go:356] "Generic (PLEG): container finished" podID="f22aa9d0-11be-4971-b00e-09405d355e86" containerID="5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7" exitCode=0 Nov 04 00:45:20.814379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.813957 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" Nov 04 00:45:20.814379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.813989 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" event={"ID":"f22aa9d0-11be-4971-b00e-09405d355e86","Type":"ContainerDied","Data":"5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7"} Nov 04 00:45:20.814379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.814071 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8" event={"ID":"f22aa9d0-11be-4971-b00e-09405d355e86","Type":"ContainerDied","Data":"f0bb83be31cfa26a80cf5f55a81dd873e87e60183e6d14685e1a560cc8f3b8ca"} Nov 04 00:45:20.814379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.814107 6208 scope.go:122] "RemoveContainer" containerID="5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7" Nov 04 00:45:20.815698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.815672 6208 generic.go:356] "Generic (PLEG): container finished" podID="4e5af56c-4637-45cd-a84d-b2e5896b082c" containerID="43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad" exitCode=0 Nov 04 00:45:20.815829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.815725 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" event={"ID":"4e5af56c-4637-45cd-a84d-b2e5896b082c","Type":"ContainerDied","Data":"43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad"} Nov 04 00:45:20.815829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.815756 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" event={"ID":"4e5af56c-4637-45cd-a84d-b2e5896b082c","Type":"ContainerDied","Data":"5346628160f55f2cd1745ef29d194bb3613e1968b3fd9aa289dbcf12d206f216"} Nov 04 00:45:20.815829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.815814 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6" Nov 04 00:45:20.827178 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.827159 6208 scope.go:122] "RemoveContainer" containerID="5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7" Nov 04 00:45:20.827559 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:45:20.827529 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7\": container with ID starting with 5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7 not found: ID does not exist" containerID="5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7" Nov 04 00:45:20.827707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.827569 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7"} err="failed to get container status \"5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7\": rpc error: code = NotFound desc = could not find container \"5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7\": container with ID starting with 5e3d88275fa1181a4d95e05b2b71f49f61b8b5ec1f16acc8fe7096c853f5e9d7 not found: ID does not exist" Nov 04 00:45:20.827707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.827593 6208 scope.go:122] "RemoveContainer" containerID="43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad" Nov 04 00:45:20.834109 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834087 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd"] Nov 04 00:45:20.834462 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834447 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4e5af56c-4637-45cd-a84d-b2e5896b082c" containerName="payload-pre-processing" Nov 04 00:45:20.834462 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834462 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5af56c-4637-45cd-a84d-b2e5896b082c" containerName="payload-pre-processing" Nov 04 00:45:20.834603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834476 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f22aa9d0-11be-4971-b00e-09405d355e86" containerName="payload-processing" Nov 04 00:45:20.834603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834483 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22aa9d0-11be-4971-b00e-09405d355e86" containerName="payload-processing" Nov 04 00:45:20.834603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834491 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1162c916-4c68-4361-b7b2-963902ab586a" containerName="cleanup" Nov 04 00:45:20.834603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834496 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1162c916-4c68-4361-b7b2-963902ab586a" containerName="cleanup" Nov 04 00:45:20.834603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834514 6208 cpu_manager.go:414] "containerMap: removing container" podUID="2725250c-844d-4435-8070-7ddfc9841238" containerName="maas-api" Nov 04 00:45:20.834603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834519 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2725250c-844d-4435-8070-7ddfc9841238" containerName="maas-api" Nov 04 00:45:20.834603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834526 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1162c916-4c68-4361-b7b2-963902ab586a" containerName="cleanup" Nov 04 00:45:20.834603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834531 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1162c916-4c68-4361-b7b2-963902ab586a" containerName="cleanup" Nov 04 00:45:20.834603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834604 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5af56c-4637-45cd-a84d-b2e5896b082c" containerName="payload-pre-processing" Nov 04 00:45:20.835080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834615 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22aa9d0-11be-4971-b00e-09405d355e86" containerName="payload-processing" Nov 04 00:45:20.835080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834620 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1162c916-4c68-4361-b7b2-963902ab586a" containerName="cleanup" Nov 04 00:45:20.835080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834625 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="2725250c-844d-4435-8070-7ddfc9841238" containerName="maas-api" Nov 04 00:45:20.835080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834631 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1162c916-4c68-4361-b7b2-963902ab586a" containerName="cleanup" Nov 04 00:45:20.835080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.834639 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1162c916-4c68-4361-b7b2-963902ab586a" containerName="cleanup" Nov 04 00:45:20.838394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.838375 6208 scope.go:122] "RemoveContainer" containerID="43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad" Nov 04 00:45:20.838813 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:45:20.838794 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad\": container with ID starting with 43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad not found: ID does not exist" containerID="43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad" Nov 04 00:45:20.838883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.838819 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad"} err="failed to get container status \"43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad\": rpc error: code = NotFound desc = could not find container \"43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad\": container with ID starting with 43422ae1ebcb545278b7ea55ef3cdcc7f2ea71d7c02c010b465e2da5058e86ad not found: ID does not exist" Nov 04 00:45:20.838979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.838959 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:20.844146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.844120 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8"] Nov 04 00:45:20.851350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.851326 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-189c01fa-kserve-self-signed-certs\"" Nov 04 00:45:20.851552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.851533 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd"] Nov 04 00:45:20.855957 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.855926 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca4fbe-86db447cc2k9j8"] Nov 04 00:45:20.890402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.890373 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6"] Nov 04 00:45:20.892915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.892896 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:20.896315 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.896291 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca4fbe-7ff85nlcw6"] Nov 04 00:45:20.956963 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.956930 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-dshm\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:20.957146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.956968 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37983280-8dad-44eb-b618-b1675e8e31ea-tls-certs\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:20.957146 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.957115 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-model-cache\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:20.957232 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.957175 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrvq\" (UniqueName: \"kubernetes.io/projected/37983280-8dad-44eb-b618-b1675e8e31ea-kube-api-access-gkrvq\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:20.957279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.957264 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-tmp-dir\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:20.957324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:20.957310 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-home\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.058418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.058330 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-model-cache\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.058418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.058371 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrvq\" (UniqueName: \"kubernetes.io/projected/37983280-8dad-44eb-b618-b1675e8e31ea-kube-api-access-gkrvq\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.058667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.058437 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-tmp-dir\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.058667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.058481 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-home\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.058667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.058515 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-dshm\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.058667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.058542 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37983280-8dad-44eb-b618-b1675e8e31ea-tls-certs\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.058889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.058830 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-tmp-dir\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.058889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.058872 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-home\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.058889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.058831 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-model-cache\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.060848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.060825 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-dshm\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.061012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.060995 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37983280-8dad-44eb-b618-b1675e8e31ea-tls-certs\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.070833 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.070809 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrvq\" (UniqueName: \"kubernetes.io/projected/37983280-8dad-44eb-b618-b1675e8e31ea-kube-api-access-gkrvq\") pod \"e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.152898 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.152866 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:21.301497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.301462 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5af56c-4637-45cd-a84d-b2e5896b082c" path="/var/lib/kubelet/pods/4e5af56c-4637-45cd-a84d-b2e5896b082c/volumes" Nov 04 00:45:21.301880 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.301862 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22aa9d0-11be-4971-b00e-09405d355e86" path="/var/lib/kubelet/pods/f22aa9d0-11be-4971-b00e-09405d355e86/volumes" Nov 04 00:45:21.431616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.431452 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd"] Nov 04 00:45:21.434650 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:45:21.434620 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37983280_8dad_44eb_b618_b1675e8e31ea.slice/crio-a81d066689edb608d41d62a3090164ed973afa3793af6eaf7438d7fd5c0aa1f5 WatchSource:0}: Error finding container a81d066689edb608d41d62a3090164ed973afa3793af6eaf7438d7fd5c0aa1f5: Status 404 returned error can't find the container with id a81d066689edb608d41d62a3090164ed973afa3793af6eaf7438d7fd5c0aa1f5 Nov 04 00:45:21.824140 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.824097 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" event={"ID":"37983280-8dad-44eb-b618-b1675e8e31ea","Type":"ContainerStarted","Data":"9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b"} Nov 04 00:45:21.824613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.824145 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" event={"ID":"37983280-8dad-44eb-b618-b1675e8e31ea","Type":"ContainerStarted","Data":"a81d066689edb608d41d62a3090164ed973afa3793af6eaf7438d7fd5c0aa1f5"} Nov 04 00:45:21.846972 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:21.846920 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" podStartSLOduration=1.846907034 podStartE2EDuration="1.846907034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:45:21.845565378 +0000 UTC m=+4540.668161541" watchObservedRunningTime="2026-11-04 00:45:21.846907034 +0000 UTC m=+4540.669503196" Nov 04 00:45:22.153736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:22.153647 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:24.419056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:24.418563 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:31.153437 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:31.153375 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:31.166226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:31.166195 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:31.894352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:31.894318 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:45:35.703093 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:45:35.703050 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:45:37.155780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.155748 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5"] Nov 04 00:45:37.156215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.156182 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1162c916-4c68-4361-b7b2-963902ab586a" containerName="cleanup" Nov 04 00:45:37.156215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.156198 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1162c916-4c68-4361-b7b2-963902ab586a" containerName="cleanup" Nov 04 00:45:37.159874 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.159853 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.167727 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.167700 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-189c01fa-kserve-self-signed-certs\"" Nov 04 00:45:37.170131 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.170100 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5"] Nov 04 00:45:37.203934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.203902 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:45:37.241670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.241638 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tls-certs\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.241807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.241679 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-dshm\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.241807 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.241697 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-home\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.241902 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.241824 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmlr\" (UniqueName: \"kubernetes.io/projected/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-kube-api-access-8qmlr\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.241902 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.241861 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tmp-dir\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.241902 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.241877 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-model-cache\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.343444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.343404 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmlr\" (UniqueName: \"kubernetes.io/projected/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-kube-api-access-8qmlr\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.343444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.343451 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tmp-dir\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.343722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.343469 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-model-cache\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.343722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.343537 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tls-certs\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.343722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.343562 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-dshm\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.343722 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.343588 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-home\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.343949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.343904 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-model-cache\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.344008 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.343952 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tmp-dir\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.344092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.344006 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-home\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.345883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.345855 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-dshm\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.346067 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.346046 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tls-certs\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.356153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.356133 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmlr\" (UniqueName: \"kubernetes.io/projected/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-kube-api-access-8qmlr\") pod \"e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.474951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.474914 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:37.751408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.751378 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5"] Nov 04 00:45:37.752795 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:45:37.752766 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3fb399c_b2c0_454c_b6e0_7e10fcee8f1c.slice/crio-96cd4b0756888ccd01d1d712cd019f301df828bf0df2bcc5bc2648a89bf9ae89 WatchSource:0}: Error finding container 96cd4b0756888ccd01d1d712cd019f301df828bf0df2bcc5bc2648a89bf9ae89: Status 404 returned error can't find the container with id 96cd4b0756888ccd01d1d712cd019f301df828bf0df2bcc5bc2648a89bf9ae89 Nov 04 00:45:37.914887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.914850 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" event={"ID":"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c","Type":"ContainerStarted","Data":"0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3"} Nov 04 00:45:37.914887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.914892 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" event={"ID":"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c","Type":"ContainerStarted","Data":"96cd4b0756888ccd01d1d712cd019f301df828bf0df2bcc5bc2648a89bf9ae89"} Nov 04 00:45:37.934208 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:37.934165 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" podStartSLOduration=0.93415045 podStartE2EDuration="934.15045ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:45:37.934029786 +0000 UTC m=+4556.756625947" watchObservedRunningTime="2026-11-04 00:45:37.93415045 +0000 UTC m=+4556.756746613" Nov 04 00:45:38.475192 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:38.475158 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:41.029176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:41.029142 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2"] Nov 04 00:45:41.032204 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:41.032181 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gltm2"] Nov 04 00:45:41.317113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:41.308681 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6c66e2-c1aa-4230-ba7d-dc61b26086d2" path="/var/lib/kubelet/pods/4c6c66e2-c1aa-4230-ba7d-dc61b26086d2/volumes" Nov 04 00:45:47.002964 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:47.002936 6208 scope.go:122] "RemoveContainer" containerID="f2ab63468845e8fc23cd2ab95a6cd4a78fbe9bbe66ead1436724a4ac4647ce20" Nov 04 00:45:47.012303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:47.012282 6208 scope.go:122] "RemoveContainer" containerID="1c0192c5e57d120da7c14a958501e15d7a685fb662d946fa94f33df377e24323" Nov 04 00:45:47.475961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:47.475928 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:47.488265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:47.488241 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:47.992828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:47.992798 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:45:53.466647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.466604 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws"] Nov 04 00:45:53.470081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.470057 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.478239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.477079 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:45:53.479013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.478979 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:45:53.479647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.479003 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:45:53.479647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.479168 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:45:53.481796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.481774 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws"] Nov 04 00:45:53.551613 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.551574 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rljws\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.551804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.551649 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rljws\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.551804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.551745 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j2bb\" (UniqueName: \"kubernetes.io/projected/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-kube-api-access-5j2bb\") pod \"maas-api-revoke-sub-452b6d7b-rljws\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.653025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.652987 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rljws\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.653253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.653083 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j2bb\" (UniqueName: \"kubernetes.io/projected/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-kube-api-access-5j2bb\") pod \"maas-api-revoke-sub-452b6d7b-rljws\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.653253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.653162 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rljws\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.653676 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.653644 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rljws\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.655568 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.655543 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rljws\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.665808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.665780 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j2bb\" (UniqueName: \"kubernetes.io/projected/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-kube-api-access-5j2bb\") pod \"maas-api-revoke-sub-452b6d7b-rljws\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:53.784919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:53.784838 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:54.059689 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:54.059664 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws"] Nov 04 00:45:54.062180 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:45:54.062149 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e42d9c_c65a_4eab_8bc7_e2b129811a38.slice/crio-aebaa5100fea674df47665ee828644a65bbb8e9b284e25f72a6e59956b648b4d WatchSource:0}: Error finding container aebaa5100fea674df47665ee828644a65bbb8e9b284e25f72a6e59956b648b4d: Status 404 returned error can't find the container with id aebaa5100fea674df47665ee828644a65bbb8e9b284e25f72a6e59956b648b4d Nov 04 00:45:55.026492 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:55.026454 6208 generic.go:356] "Generic (PLEG): container finished" podID="d6e42d9c-c65a-4eab-8bc7-e2b129811a38" containerID="f47a9ff62ff0e13dfb8600438215fdefdea3d7f4a6ee5a4279c7dde0565a450b" exitCode=0 Nov 04 00:45:55.026896 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:55.026537 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" event={"ID":"d6e42d9c-c65a-4eab-8bc7-e2b129811a38","Type":"ContainerDied","Data":"f47a9ff62ff0e13dfb8600438215fdefdea3d7f4a6ee5a4279c7dde0565a450b"} Nov 04 00:45:55.026896 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:55.026602 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" event={"ID":"d6e42d9c-c65a-4eab-8bc7-e2b129811a38","Type":"ContainerStarted","Data":"aebaa5100fea674df47665ee828644a65bbb8e9b284e25f72a6e59956b648b4d"} Nov 04 00:45:56.333770 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.333387 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:56.425803 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.425768 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-kube-api-access-5j2bb\" (UniqueName: \"kubernetes.io/projected/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-kube-api-access-5j2bb\") pod \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " Nov 04 00:45:56.426387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.426359 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-xks-ca\") pod \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " Nov 04 00:45:56.426602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.426581 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-openshift-ca\") pod \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\" (UID: \"d6e42d9c-c65a-4eab-8bc7-e2b129811a38\") " Nov 04 00:45:56.427501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.427473 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-openshift-ca" pod "d6e42d9c-c65a-4eab-8bc7-e2b129811a38" (UID: "d6e42d9c-c65a-4eab-8bc7-e2b129811a38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:45:56.428514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.428463 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-kube-api-access-5j2bb" pod "d6e42d9c-c65a-4eab-8bc7-e2b129811a38" (UID: "d6e42d9c-c65a-4eab-8bc7-e2b129811a38"). InnerVolumeSpecName "kube-api-access-5j2bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:45:56.429621 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.429587 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-xks-ca" pod "d6e42d9c-c65a-4eab-8bc7-e2b129811a38" (UID: "d6e42d9c-c65a-4eab-8bc7-e2b129811a38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:45:56.531750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.531704 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:56.533128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.533089 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j2bb\" (UniqueName: \"kubernetes.io/projected/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-kube-api-access-5j2bb\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:56.533128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:56.533129 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e42d9c-c65a-4eab-8bc7-e2b129811a38-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:45:57.037643 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:57.037606 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg"] Nov 04 00:45:57.037934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:57.037913 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4t2rg"] Nov 04 00:45:57.047897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:57.047643 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" Nov 04 00:45:57.047897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:57.047718 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws" event={"ID":"d6e42d9c-c65a-4eab-8bc7-e2b129811a38","Type":"ContainerDied","Data":"aebaa5100fea674df47665ee828644a65bbb8e9b284e25f72a6e59956b648b4d"} Nov 04 00:45:57.047897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:57.047777 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aebaa5100fea674df47665ee828644a65bbb8e9b284e25f72a6e59956b648b4d" Nov 04 00:45:57.342512 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:57.342352 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a71493ff-e4e2-4948-a1c0-34fcc30b663d" path="/var/lib/kubelet/pods/a71493ff-e4e2-4948-a1c0-34fcc30b663d/volumes" Nov 04 00:45:58.529714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:45:58.529677 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:46:04.346273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:04.346236 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd"] Nov 04 00:46:04.390423 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:04.390387 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:46:05.060411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:05.060376 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:46:05.704289 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:46:05.704250 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:46:06.334438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:06.334374 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5"] Nov 04 00:46:06.371483 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:06.371444 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:46:07.028917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:07.028852 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69"] Nov 04 00:46:07.032001 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:07.031967 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d9z69"] Nov 04 00:46:07.335113 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:07.317339 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407605d5-b8a1-488c-8d3a-7f9b5ad59a5a" path="/var/lib/kubelet/pods/407605d5-b8a1-488c-8d3a-7f9b5ad59a5a/volumes" Nov 04 00:46:17.878253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.878209 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k"] Nov 04 00:46:17.878817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.878635 6208 cpu_manager.go:414] "containerMap: removing container" podUID="d6e42d9c-c65a-4eab-8bc7-e2b129811a38" containerName="revoke-keys" Nov 04 00:46:17.878817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.878649 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e42d9c-c65a-4eab-8bc7-e2b129811a38" containerName="revoke-keys" Nov 04 00:46:17.878817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.878729 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e42d9c-c65a-4eab-8bc7-e2b129811a38" containerName="revoke-keys" Nov 04 00:46:17.882119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.882091 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:17.890562 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.890536 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:46:17.890688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.890617 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:46:17.890688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.890650 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:46:17.890804 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.890704 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:46:17.891352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.891326 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k"] Nov 04 00:46:17.988995 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.988957 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jsq8k\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:17.989191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.989092 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47v5h\" (UniqueName: \"kubernetes.io/projected/4bd1d263-012f-4dbf-babc-b0af5b450ea3-kube-api-access-47v5h\") pod \"maas-api-revoke-sub-3e31a506-jsq8k\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:17.989191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:17.989160 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jsq8k\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:18.090237 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:18.090190 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jsq8k\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:18.090428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:18.090305 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jsq8k\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:18.090428 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:18.090369 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47v5h\" (UniqueName: \"kubernetes.io/projected/4bd1d263-012f-4dbf-babc-b0af5b450ea3-kube-api-access-47v5h\") pod \"maas-api-revoke-sub-3e31a506-jsq8k\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:18.091012 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:18.090987 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jsq8k\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:18.092641 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:18.092612 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jsq8k\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:18.103070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:18.103046 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47v5h\" (UniqueName: \"kubernetes.io/projected/4bd1d263-012f-4dbf-babc-b0af5b450ea3-kube-api-access-47v5h\") pod \"maas-api-revoke-sub-3e31a506-jsq8k\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:18.198381 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:18.198311 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:18.472937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:18.472893 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k"] Nov 04 00:46:18.475529 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:46:18.475491 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd1d263_012f_4dbf_babc_b0af5b450ea3.slice/crio-f45c6bf030bc0ec8eeebb49f40e380f0a3d85a6e15c426063c8f2f4794d8b48f WatchSource:0}: Error finding container f45c6bf030bc0ec8eeebb49f40e380f0a3d85a6e15c426063c8f2f4794d8b48f: Status 404 returned error can't find the container with id f45c6bf030bc0ec8eeebb49f40e380f0a3d85a6e15c426063c8f2f4794d8b48f Nov 04 00:46:19.192532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.192436 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" event={"ID":"4bd1d263-012f-4dbf-babc-b0af5b450ea3","Type":"ContainerStarted","Data":"11e80bb659221d68a1d09374a9d5b23e140647b0f10f26f36bd7c28a16910329"} Nov 04 00:46:19.192532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.192499 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" event={"ID":"4bd1d263-012f-4dbf-babc-b0af5b450ea3","Type":"ContainerStarted","Data":"f45c6bf030bc0ec8eeebb49f40e380f0a3d85a6e15c426063c8f2f4794d8b48f"} Nov 04 00:46:19.213921 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.213864 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" podStartSLOduration=2.213850525 podStartE2EDuration="2.213850525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:46:19.211832589 +0000 UTC m=+4598.034428752" watchObservedRunningTime="2026-11-04 00:46:19.213850525 +0000 UTC m=+4598.036446687" Nov 04 00:46:19.366207 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.366167 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" podUID="37983280-8dad-44eb-b618-b1675e8e31ea" containerName="main" containerID="cri-o://9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b" gracePeriod=45 Nov 04 00:46:19.626691 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.626665 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:46:19.708537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.708467 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-home\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-home\") pod \"37983280-8dad-44eb-b618-b1675e8e31ea\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " Nov 04 00:46:19.708537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.708513 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-tmp-dir\") pod \"37983280-8dad-44eb-b618-b1675e8e31ea\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " Nov 04 00:46:19.708759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.708621 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37983280-8dad-44eb-b618-b1675e8e31ea-kube-api-access-gkrvq\" (UniqueName: \"kubernetes.io/projected/37983280-8dad-44eb-b618-b1675e8e31ea-kube-api-access-gkrvq\") pod \"37983280-8dad-44eb-b618-b1675e8e31ea\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " Nov 04 00:46:19.708759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.708666 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-dshm\") pod \"37983280-8dad-44eb-b618-b1675e8e31ea\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " Nov 04 00:46:19.708759 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.708696 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-model-cache\") pod \"37983280-8dad-44eb-b618-b1675e8e31ea\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " Nov 04 00:46:19.708927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.708764 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-home" pod "37983280-8dad-44eb-b618-b1675e8e31ea" (UID: "37983280-8dad-44eb-b618-b1675e8e31ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:46:19.708927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.708806 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37983280-8dad-44eb-b618-b1675e8e31ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/37983280-8dad-44eb-b618-b1675e8e31ea-tls-certs\") pod \"37983280-8dad-44eb-b618-b1675e8e31ea\" (UID: \"37983280-8dad-44eb-b618-b1675e8e31ea\") " Nov 04 00:46:19.708927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.708829 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-tmp-dir" pod "37983280-8dad-44eb-b618-b1675e8e31ea" (UID: "37983280-8dad-44eb-b618-b1675e8e31ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:46:19.708927 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.708920 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-model-cache" pod "37983280-8dad-44eb-b618-b1675e8e31ea" (UID: "37983280-8dad-44eb-b618-b1675e8e31ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:46:19.709193 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.709172 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:19.709251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.709197 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:19.709251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.709206 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:19.710842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.710814 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-dshm" pod "37983280-8dad-44eb-b618-b1675e8e31ea" (UID: "37983280-8dad-44eb-b618-b1675e8e31ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:46:19.710991 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.710844 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37983280-8dad-44eb-b618-b1675e8e31ea-kube-api-access-gkrvq" pod "37983280-8dad-44eb-b618-b1675e8e31ea" (UID: "37983280-8dad-44eb-b618-b1675e8e31ea"). InnerVolumeSpecName "kube-api-access-gkrvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:46:19.710991 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.710845 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37983280-8dad-44eb-b618-b1675e8e31ea-tls-certs" pod "37983280-8dad-44eb-b618-b1675e8e31ea" (UID: "37983280-8dad-44eb-b618-b1675e8e31ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:46:19.810255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.810179 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkrvq\" (UniqueName: \"kubernetes.io/projected/37983280-8dad-44eb-b618-b1675e8e31ea-kube-api-access-gkrvq\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:19.810255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.810206 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37983280-8dad-44eb-b618-b1675e8e31ea-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:19.810255 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:19.810215 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37983280-8dad-44eb-b618-b1675e8e31ea-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:20.199318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.199280 6208 generic.go:356] "Generic (PLEG): container finished" podID="4bd1d263-012f-4dbf-babc-b0af5b450ea3" containerID="11e80bb659221d68a1d09374a9d5b23e140647b0f10f26f36bd7c28a16910329" exitCode=0 Nov 04 00:46:20.199760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.199365 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" event={"ID":"4bd1d263-012f-4dbf-babc-b0af5b450ea3","Type":"ContainerDied","Data":"11e80bb659221d68a1d09374a9d5b23e140647b0f10f26f36bd7c28a16910329"} Nov 04 00:46:20.200696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.200673 6208 generic.go:356] "Generic (PLEG): container finished" podID="37983280-8dad-44eb-b618-b1675e8e31ea" containerID="9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b" exitCode=0 Nov 04 00:46:20.200827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.200706 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" Nov 04 00:46:20.200827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.200758 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" event={"ID":"37983280-8dad-44eb-b618-b1675e8e31ea","Type":"ContainerDied","Data":"9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b"} Nov 04 00:46:20.200827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.200803 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd" event={"ID":"37983280-8dad-44eb-b618-b1675e8e31ea","Type":"ContainerDied","Data":"a81d066689edb608d41d62a3090164ed973afa3793af6eaf7438d7fd5c0aa1f5"} Nov 04 00:46:20.201008 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.200825 6208 scope.go:122] "RemoveContainer" containerID="9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b" Nov 04 00:46:20.210492 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.210476 6208 scope.go:122] "RemoveContainer" containerID="9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b" Nov 04 00:46:20.210754 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:46:20.210736 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b\": container with ID starting with 9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b not found: ID does not exist" containerID="9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b" Nov 04 00:46:20.210794 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.210764 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b"} err="failed to get container status \"9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b\": rpc error: code = NotFound desc = could not find container \"9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b\": container with ID starting with 9df16a314d7ba8e3f272de5fab79840f328bcabf6223f8b02c08f78c42b4a40b not found: ID does not exist" Nov 04 00:46:20.233596 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.233571 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd"] Nov 04 00:46:20.237748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:20.237721 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-189c01fa-kserve-75fbc8466f-t94hd"] Nov 04 00:46:21.293123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.293095 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37983280-8dad-44eb-b618-b1675e8e31ea" path="/var/lib/kubelet/pods/37983280-8dad-44eb-b618-b1675e8e31ea/volumes" Nov 04 00:46:21.335331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.335306 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:21.357310 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.357240 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" podUID="e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" containerName="main" containerID="cri-o://0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3" gracePeriod=45 Nov 04 00:46:21.424219 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.424195 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd1d263-012f-4dbf-babc-b0af5b450ea3-kube-api-access-47v5h\" (UniqueName: \"kubernetes.io/projected/4bd1d263-012f-4dbf-babc-b0af5b450ea3-kube-api-access-47v5h\") pod \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " Nov 04 00:46:21.424378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.424310 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-openshift-ca\") pod \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " Nov 04 00:46:21.424378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.424342 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-xks-ca\") pod \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\" (UID: \"4bd1d263-012f-4dbf-babc-b0af5b450ea3\") " Nov 04 00:46:21.424727 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.424698 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-openshift-ca" pod "4bd1d263-012f-4dbf-babc-b0af5b450ea3" (UID: "4bd1d263-012f-4dbf-babc-b0af5b450ea3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:46:21.426368 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.426344 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-xks-ca" pod "4bd1d263-012f-4dbf-babc-b0af5b450ea3" (UID: "4bd1d263-012f-4dbf-babc-b0af5b450ea3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:46:21.426478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.426366 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd1d263-012f-4dbf-babc-b0af5b450ea3-kube-api-access-47v5h" pod "4bd1d263-012f-4dbf-babc-b0af5b450ea3" (UID: "4bd1d263-012f-4dbf-babc-b0af5b450ea3"). InnerVolumeSpecName "kube-api-access-47v5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:46:21.525583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.525558 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47v5h\" (UniqueName: \"kubernetes.io/projected/4bd1d263-012f-4dbf-babc-b0af5b450ea3-kube-api-access-47v5h\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:21.525673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.525586 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:21.525673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.525605 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bd1d263-012f-4dbf-babc-b0af5b450ea3-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:21.602412 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.602384 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:46:21.727475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.727383 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-dshm\") pod \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " Nov 04 00:46:21.727475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.727432 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tls-certs\") pod \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " Nov 04 00:46:21.727716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.727480 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tmp-dir\") pod \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " Nov 04 00:46:21.727716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.727500 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-kube-api-access-8qmlr\" (UniqueName: \"kubernetes.io/projected/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-kube-api-access-8qmlr\") pod \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " Nov 04 00:46:21.727716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.727528 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-home\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-home\") pod \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " Nov 04 00:46:21.727716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.727596 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-model-cache\") pod \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\" (UID: \"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c\") " Nov 04 00:46:21.727923 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.727795 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tmp-dir" pod "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" (UID: "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:46:21.727995 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.727971 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-home" pod "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" (UID: "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:46:21.728080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.727982 6208 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tmp-dir\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:21.728227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.728189 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-model-cache" pod "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" (UID: "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:46:21.729764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.729738 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-dshm" pod "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" (UID: "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:46:21.730081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.730059 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tls-certs" pod "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" (UID: "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:46:21.730281 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.730262 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-kube-api-access-8qmlr" pod "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" (UID: "e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c"). InnerVolumeSpecName "kube-api-access-8qmlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:46:21.829257 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.829221 6208 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-dshm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:21.829257 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.829251 6208 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-tls-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:21.829257 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.829261 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qmlr\" (UniqueName: \"kubernetes.io/projected/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-kube-api-access-8qmlr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:21.829498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.829271 6208 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-home\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:21.829498 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:21.829281 6208 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c-model-cache\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:22.216657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.216623 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" Nov 04 00:46:22.216657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.216630 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k" event={"ID":"4bd1d263-012f-4dbf-babc-b0af5b450ea3","Type":"ContainerDied","Data":"f45c6bf030bc0ec8eeebb49f40e380f0a3d85a6e15c426063c8f2f4794d8b48f"} Nov 04 00:46:22.216906 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.216724 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f45c6bf030bc0ec8eeebb49f40e380f0a3d85a6e15c426063c8f2f4794d8b48f" Nov 04 00:46:22.218075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.218049 6208 generic.go:356] "Generic (PLEG): container finished" podID="e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" containerID="0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3" exitCode=0 Nov 04 00:46:22.218233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.218105 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" Nov 04 00:46:22.218233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.218147 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" event={"ID":"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c","Type":"ContainerDied","Data":"0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3"} Nov 04 00:46:22.218233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.218188 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5" event={"ID":"e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c","Type":"ContainerDied","Data":"96cd4b0756888ccd01d1d712cd019f301df828bf0df2bcc5bc2648a89bf9ae89"} Nov 04 00:46:22.218233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.218208 6208 scope.go:122] "RemoveContainer" containerID="0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3" Nov 04 00:46:22.227578 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.227560 6208 scope.go:122] "RemoveContainer" containerID="0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3" Nov 04 00:46:22.227931 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:46:22.227831 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3\": container with ID starting with 0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3 not found: ID does not exist" containerID="0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3" Nov 04 00:46:22.227931 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.227868 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3"} err="failed to get container status \"0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3\": rpc error: code = NotFound desc = could not find container \"0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3\": container with ID starting with 0648788bdbe1a954fbcdd44994c70f471ad5cdb1f259a9346c1895f5554762d3 not found: ID does not exist" Nov 04 00:46:22.246357 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.246335 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5"] Nov 04 00:46:22.253988 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:22.253968 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-189c01fa-kserve-57b4667485-tbxs5"] Nov 04 00:46:23.293320 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:23.293279 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" path="/var/lib/kubelet/pods/e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c/volumes" Nov 04 00:46:24.486733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:24.486702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:46:30.665226 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:30.665152 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:46:30.665905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:30.665871 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:46:35.499689 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.499638 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh"] Nov 04 00:46:35.500244 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.500221 6208 cpu_manager.go:414] "containerMap: removing container" podUID="37983280-8dad-44eb-b618-b1675e8e31ea" containerName="main" Nov 04 00:46:35.500375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.500247 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37983280-8dad-44eb-b618-b1675e8e31ea" containerName="main" Nov 04 00:46:35.500375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.500267 6208 cpu_manager.go:414] "containerMap: removing container" podUID="e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" containerName="main" Nov 04 00:46:35.500375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.500274 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" containerName="main" Nov 04 00:46:35.500375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.500324 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4bd1d263-012f-4dbf-babc-b0af5b450ea3" containerName="revoke-keys" Nov 04 00:46:35.500375 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.500335 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd1d263-012f-4dbf-babc-b0af5b450ea3" containerName="revoke-keys" Nov 04 00:46:35.500648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.500448 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd1d263-012f-4dbf-babc-b0af5b450ea3" containerName="revoke-keys" Nov 04 00:46:35.500648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.500464 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="37983280-8dad-44eb-b618-b1675e8e31ea" containerName="main" Nov 04 00:46:35.500648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.500476 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3fb399c-b2c0-454c-b6e0-7e10fcee8f1c" containerName="main" Nov 04 00:46:35.505214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.505190 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.513187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.513152 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh"] Nov 04 00:46:35.514094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.513436 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:46:35.514094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.513549 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:46:35.514094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.513575 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:46:35.514094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.513655 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:46:35.646358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.646314 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfqm\" (UniqueName: \"kubernetes.io/projected/0b4d1e80-2803-4006-834e-96f745f91016-kube-api-access-pwfqm\") pod \"maas-api-revoke-sub-0fd409a3-rs9rh\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.646541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.646364 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd409a3-rs9rh\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.646600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.646570 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd409a3-rs9rh\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.704568 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:46:35.704524 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:46:35.748056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.748014 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd409a3-rs9rh\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.748238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.748120 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfqm\" (UniqueName: \"kubernetes.io/projected/0b4d1e80-2803-4006-834e-96f745f91016-kube-api-access-pwfqm\") pod \"maas-api-revoke-sub-0fd409a3-rs9rh\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.748238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.748159 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd409a3-rs9rh\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.748618 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.748598 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd409a3-rs9rh\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.750403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.750342 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd409a3-rs9rh\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.759970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.759948 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfqm\" (UniqueName: \"kubernetes.io/projected/0b4d1e80-2803-4006-834e-96f745f91016-kube-api-access-pwfqm\") pod \"maas-api-revoke-sub-0fd409a3-rs9rh\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:35.822778 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:35.822755 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:36.100464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:36.100432 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh"] Nov 04 00:46:36.102134 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:46:36.102097 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4d1e80_2803_4006_834e_96f745f91016.slice/crio-26fd70eafd6ef94a0e5080a4978402f5ad909518dadfc097ef7f3d14bf0759d5 WatchSource:0}: Error finding container 26fd70eafd6ef94a0e5080a4978402f5ad909518dadfc097ef7f3d14bf0759d5: Status 404 returned error can't find the container with id 26fd70eafd6ef94a0e5080a4978402f5ad909518dadfc097ef7f3d14bf0759d5 Nov 04 00:46:36.343294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:36.343256 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" event={"ID":"0b4d1e80-2803-4006-834e-96f745f91016","Type":"ContainerStarted","Data":"26fd70eafd6ef94a0e5080a4978402f5ad909518dadfc097ef7f3d14bf0759d5"} Nov 04 00:46:37.349587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:37.349546 6208 generic.go:356] "Generic (PLEG): container finished" podID="0b4d1e80-2803-4006-834e-96f745f91016" containerID="5b395421803e351108693cde47ee533e1b8bd0d7ca12de6e203224ba21c21da9" exitCode=0 Nov 04 00:46:37.350020 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:37.349634 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" event={"ID":"0b4d1e80-2803-4006-834e-96f745f91016","Type":"ContainerDied","Data":"5b395421803e351108693cde47ee533e1b8bd0d7ca12de6e203224ba21c21da9"} Nov 04 00:46:38.484277 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.484249 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:38.673348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.673262 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-openshift-ca\") pod \"0b4d1e80-2803-4006-834e-96f745f91016\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " Nov 04 00:46:38.673348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.673320 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-xks-ca\") pod \"0b4d1e80-2803-4006-834e-96f745f91016\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " Nov 04 00:46:38.673348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.673349 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4d1e80-2803-4006-834e-96f745f91016-kube-api-access-pwfqm\" (UniqueName: \"kubernetes.io/projected/0b4d1e80-2803-4006-834e-96f745f91016-kube-api-access-pwfqm\") pod \"0b4d1e80-2803-4006-834e-96f745f91016\" (UID: \"0b4d1e80-2803-4006-834e-96f745f91016\") " Nov 04 00:46:38.673709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.673679 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-openshift-ca" pod "0b4d1e80-2803-4006-834e-96f745f91016" (UID: "0b4d1e80-2803-4006-834e-96f745f91016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:46:38.675486 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.675458 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-xks-ca" pod "0b4d1e80-2803-4006-834e-96f745f91016" (UID: "0b4d1e80-2803-4006-834e-96f745f91016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:46:38.675624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.675553 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4d1e80-2803-4006-834e-96f745f91016-kube-api-access-pwfqm" pod "0b4d1e80-2803-4006-834e-96f745f91016" (UID: "0b4d1e80-2803-4006-834e-96f745f91016"). InnerVolumeSpecName "kube-api-access-pwfqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:46:38.774469 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.774436 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:38.774469 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.774464 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b4d1e80-2803-4006-834e-96f745f91016-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:38.774469 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:38.774475 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfqm\" (UniqueName: \"kubernetes.io/projected/0b4d1e80-2803-4006-834e-96f745f91016-kube-api-access-pwfqm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:46:39.361545 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:39.361512 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" Nov 04 00:46:39.361728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:39.361512 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh" event={"ID":"0b4d1e80-2803-4006-834e-96f745f91016","Type":"ContainerDied","Data":"26fd70eafd6ef94a0e5080a4978402f5ad909518dadfc097ef7f3d14bf0759d5"} Nov 04 00:46:39.361728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:39.361646 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26fd70eafd6ef94a0e5080a4978402f5ad909518dadfc097ef7f3d14bf0759d5" Nov 04 00:46:47.052679 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:47.052648 6208 scope.go:122] "RemoveContainer" containerID="d8fc3acb4c0645c73c8efcc184364838577ecb6f5f38023d1a464b0230f485fd" Nov 04 00:46:47.061523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:47.061498 6208 scope.go:122] "RemoveContainer" containerID="64f8aaec3be9a41c6c80c8ce77ef717636d52c7fa0de5d99486c65995bf42002" Nov 04 00:46:58.588996 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:58.588957 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:46:58.911884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:58.911797 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:46:59.138481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.138448 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws"] Nov 04 00:46:59.138863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.138849 6208 cpu_manager.go:414] "containerMap: removing container" podUID="0b4d1e80-2803-4006-834e-96f745f91016" containerName="revoke-keys" Nov 04 00:46:59.138918 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.138866 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4d1e80-2803-4006-834e-96f745f91016" containerName="revoke-keys" Nov 04 00:46:59.138970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.138959 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4d1e80-2803-4006-834e-96f745f91016" containerName="revoke-keys" Nov 04 00:46:59.142167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.142149 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:46:59.151053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.151013 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws"] Nov 04 00:46:59.254396 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.254363 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccfpx\" (UniqueName: \"kubernetes.io/projected/a7556abd-0a59-4d61-a928-9b9eb72f7e81-kube-api-access-ccfpx\") pod \"payload-pre-processing-5cb47fbf9b-lhcws\" (UID: \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\") " pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:46:59.254559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.254399 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7556abd-0a59-4d61-a928-9b9eb72f7e81-plugins-config-volume\") pod \"payload-pre-processing-5cb47fbf9b-lhcws\" (UID: \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\") " pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:46:59.355865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.355828 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccfpx\" (UniqueName: \"kubernetes.io/projected/a7556abd-0a59-4d61-a928-9b9eb72f7e81-kube-api-access-ccfpx\") pod \"payload-pre-processing-5cb47fbf9b-lhcws\" (UID: \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\") " pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:46:59.355865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.355864 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7556abd-0a59-4d61-a928-9b9eb72f7e81-plugins-config-volume\") pod \"payload-pre-processing-5cb47fbf9b-lhcws\" (UID: \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\") " pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:46:59.356609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.356588 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7556abd-0a59-4d61-a928-9b9eb72f7e81-plugins-config-volume\") pod \"payload-pre-processing-5cb47fbf9b-lhcws\" (UID: \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\") " pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:46:59.368960 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.368928 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccfpx\" (UniqueName: \"kubernetes.io/projected/a7556abd-0a59-4d61-a928-9b9eb72f7e81-kube-api-access-ccfpx\") pod \"payload-pre-processing-5cb47fbf9b-lhcws\" (UID: \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\") " pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:46:59.456519 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.456489 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:46:59.731905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:46:59.731875 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws"] Nov 04 00:46:59.733678 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:46:59.733652 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7556abd_0a59_4d61_a928_9b9eb72f7e81.slice/crio-997ee1033c260df9e7da84da1afdb57b3ff83c5b4807cb80f5199112fd4c5ee0 WatchSource:0}: Error finding container 997ee1033c260df9e7da84da1afdb57b3ff83c5b4807cb80f5199112fd4c5ee0: Status 404 returned error can't find the container with id 997ee1033c260df9e7da84da1afdb57b3ff83c5b4807cb80f5199112fd4c5ee0 Nov 04 00:47:00.473790 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:00.473747 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" event={"ID":"a7556abd-0a59-4d61-a928-9b9eb72f7e81","Type":"ContainerStarted","Data":"3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822"} Nov 04 00:47:00.473990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:00.473807 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" event={"ID":"a7556abd-0a59-4d61-a928-9b9eb72f7e81","Type":"ContainerStarted","Data":"997ee1033c260df9e7da84da1afdb57b3ff83c5b4807cb80f5199112fd4c5ee0"} Nov 04 00:47:00.473990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:00.473876 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:47:00.494819 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:00.494766 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" podStartSLOduration=1.494752794 podStartE2EDuration="1.494752794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:47:00.492291418 +0000 UTC m=+4639.314887590" watchObservedRunningTime="2026-11-04 00:47:00.494752794 +0000 UTC m=+4639.317348955" Nov 04 00:47:05.130860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:05.130821 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:47:05.703324 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:47:05.703286 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:47:11.481632 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:11.481595 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:47:11.532652 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:11.532610 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx"] Nov 04 00:47:13.393777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.393743 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-684b8f58d7-s7fth"] Nov 04 00:47:13.397490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.397469 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:13.408377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.408348 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-684b8f58d7-s7fth"] Nov 04 00:47:13.488931 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.488894 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2430d0a-1990-4e53-8297-241b76baae8b-plugins-config-volume\") pod \"payload-processing-684b8f58d7-s7fth\" (UID: \"a2430d0a-1990-4e53-8297-241b76baae8b\") " pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:13.489145 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.489016 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9bw\" (UniqueName: \"kubernetes.io/projected/a2430d0a-1990-4e53-8297-241b76baae8b-kube-api-access-9r9bw\") pod \"payload-processing-684b8f58d7-s7fth\" (UID: \"a2430d0a-1990-4e53-8297-241b76baae8b\") " pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:13.589894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.589858 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9bw\" (UniqueName: \"kubernetes.io/projected/a2430d0a-1990-4e53-8297-241b76baae8b-kube-api-access-9r9bw\") pod \"payload-processing-684b8f58d7-s7fth\" (UID: \"a2430d0a-1990-4e53-8297-241b76baae8b\") " pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:13.590119 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.589920 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2430d0a-1990-4e53-8297-241b76baae8b-plugins-config-volume\") pod \"payload-processing-684b8f58d7-s7fth\" (UID: \"a2430d0a-1990-4e53-8297-241b76baae8b\") " pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:13.590602 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.590580 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2430d0a-1990-4e53-8297-241b76baae8b-plugins-config-volume\") pod \"payload-processing-684b8f58d7-s7fth\" (UID: \"a2430d0a-1990-4e53-8297-241b76baae8b\") " pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:13.602655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.602630 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9bw\" (UniqueName: \"kubernetes.io/projected/a2430d0a-1990-4e53-8297-241b76baae8b-kube-api-access-9r9bw\") pod \"payload-processing-684b8f58d7-s7fth\" (UID: \"a2430d0a-1990-4e53-8297-241b76baae8b\") " pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:13.712206 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.712180 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:13.993093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:13.993067 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-684b8f58d7-s7fth"] Nov 04 00:47:13.995297 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:47:13.995263 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2430d0a_1990_4e53_8297_241b76baae8b.slice/crio-b4ef2e91915ee2c79ca731679b00890dbe925c7adea1db86a0e3746ba17e0a49 WatchSource:0}: Error finding container b4ef2e91915ee2c79ca731679b00890dbe925c7adea1db86a0e3746ba17e0a49: Status 404 returned error can't find the container with id b4ef2e91915ee2c79ca731679b00890dbe925c7adea1db86a0e3746ba17e0a49 Nov 04 00:47:14.548660 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:14.548622 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" event={"ID":"a2430d0a-1990-4e53-8297-241b76baae8b","Type":"ContainerStarted","Data":"d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8"} Nov 04 00:47:14.549108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:14.548763 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:14.549108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:14.548782 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" event={"ID":"a2430d0a-1990-4e53-8297-241b76baae8b","Type":"ContainerStarted","Data":"b4ef2e91915ee2c79ca731679b00890dbe925c7adea1db86a0e3746ba17e0a49"} Nov 04 00:47:14.568748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:14.568697 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" podStartSLOduration=1.568684825 podStartE2EDuration="1.568684825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:47:14.566895969 +0000 UTC m=+4653.389492135" watchObservedRunningTime="2026-11-04 00:47:14.568684825 +0000 UTC m=+4653.391281018" Nov 04 00:47:16.533448 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:16.533407 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" podUID="a0e5a1e6-cb54-4115-ae52-3278607679c5" containerName="payload-pre-processing" containerID="cri-o://2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f" gracePeriod=25 Nov 04 00:47:16.785965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:16.785911 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 04 00:47:16.920092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:16.920054 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e5a1e6-cb54-4115-ae52-3278607679c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5a1e6-cb54-4115-ae52-3278607679c5-plugins-config-volume\") pod \"a0e5a1e6-cb54-4115-ae52-3278607679c5\" (UID: \"a0e5a1e6-cb54-4115-ae52-3278607679c5\") " Nov 04 00:47:16.920271 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:16.920250 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e5a1e6-cb54-4115-ae52-3278607679c5-kube-api-access-t9bfs\" (UniqueName: \"kubernetes.io/projected/a0e5a1e6-cb54-4115-ae52-3278607679c5-kube-api-access-t9bfs\") pod \"a0e5a1e6-cb54-4115-ae52-3278607679c5\" (UID: \"a0e5a1e6-cb54-4115-ae52-3278607679c5\") " Nov 04 00:47:16.920850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:16.920824 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e5a1e6-cb54-4115-ae52-3278607679c5-plugins-config-volume" pod "a0e5a1e6-cb54-4115-ae52-3278607679c5" (UID: "a0e5a1e6-cb54-4115-ae52-3278607679c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:47:16.922764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:16.922732 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e5a1e6-cb54-4115-ae52-3278607679c5-kube-api-access-t9bfs" pod "a0e5a1e6-cb54-4115-ae52-3278607679c5" (UID: "a0e5a1e6-cb54-4115-ae52-3278607679c5"). InnerVolumeSpecName "kube-api-access-t9bfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:47:17.021600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.021568 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e5a1e6-cb54-4115-ae52-3278607679c5-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:47:17.021600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.021599 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9bfs\" (UniqueName: \"kubernetes.io/projected/a0e5a1e6-cb54-4115-ae52-3278607679c5-kube-api-access-t9bfs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:47:17.564709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.564617 6208 generic.go:356] "Generic (PLEG): container finished" podID="a0e5a1e6-cb54-4115-ae52-3278607679c5" containerID="2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f" exitCode=0 Nov 04 00:47:17.564709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.564674 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" Nov 04 00:47:17.565260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.564705 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" event={"ID":"a0e5a1e6-cb54-4115-ae52-3278607679c5","Type":"ContainerDied","Data":"2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f"} Nov 04 00:47:17.565260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.564771 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx" event={"ID":"a0e5a1e6-cb54-4115-ae52-3278607679c5","Type":"ContainerDied","Data":"daa63e1d1f12ca378c64200a9621600c820c01c786cc55e2a2ded5d307c71d85"} Nov 04 00:47:17.565260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.564803 6208 scope.go:122] "RemoveContainer" containerID="2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f" Nov 04 00:47:17.576845 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.576825 6208 scope.go:122] "RemoveContainer" containerID="2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f" Nov 04 00:47:17.577124 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:47:17.577097 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f\": container with ID starting with 2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f not found: ID does not exist" containerID="2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f" Nov 04 00:47:17.577206 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.577125 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f"} err="failed to get container status \"2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f\": rpc error: code = NotFound desc = could not find container \"2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f\": container with ID starting with 2b08fae9fe996a0cc9b3830345cb4c2cc08fcd85b8d2a7a826c536c2d7656e5f not found: ID does not exist" Nov 04 00:47:17.583692 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.583669 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx"] Nov 04 00:47:17.587958 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:17.587934 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w2ckx"] Nov 04 00:47:19.293225 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:19.293183 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e5a1e6-cb54-4115-ae52-3278607679c5" path="/var/lib/kubelet/pods/a0e5a1e6-cb54-4115-ae52-3278607679c5/volumes" Nov 04 00:47:24.540683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:24.540644 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:47:25.556009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:25.555979 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:25.607322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:25.607283 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-4dsdb"] Nov 04 00:47:30.608137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:30.608083 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" podUID="c07df26b-e714-4555-a633-fa616bb9be2e" containerName="payload-processing" containerID="cri-o://2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db" gracePeriod=25 Nov 04 00:47:30.856845 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:30.856818 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 04 00:47:30.951904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:30.951807 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07df26b-e714-4555-a633-fa616bb9be2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07df26b-e714-4555-a633-fa616bb9be2e-plugins-config-volume\") pod \"c07df26b-e714-4555-a633-fa616bb9be2e\" (UID: \"c07df26b-e714-4555-a633-fa616bb9be2e\") " Nov 04 00:47:30.951904 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:30.951897 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07df26b-e714-4555-a633-fa616bb9be2e-kube-api-access-k65hx\" (UniqueName: \"kubernetes.io/projected/c07df26b-e714-4555-a633-fa616bb9be2e-kube-api-access-k65hx\") pod \"c07df26b-e714-4555-a633-fa616bb9be2e\" (UID: \"c07df26b-e714-4555-a633-fa616bb9be2e\") " Nov 04 00:47:30.952477 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:30.952443 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07df26b-e714-4555-a633-fa616bb9be2e-plugins-config-volume" pod "c07df26b-e714-4555-a633-fa616bb9be2e" (UID: "c07df26b-e714-4555-a633-fa616bb9be2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:47:30.955248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:30.955213 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07df26b-e714-4555-a633-fa616bb9be2e-kube-api-access-k65hx" pod "c07df26b-e714-4555-a633-fa616bb9be2e" (UID: "c07df26b-e714-4555-a633-fa616bb9be2e"). InnerVolumeSpecName "kube-api-access-k65hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:47:31.053661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.053627 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07df26b-e714-4555-a633-fa616bb9be2e-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:47:31.053661 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.053655 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65hx\" (UniqueName: \"kubernetes.io/projected/c07df26b-e714-4555-a633-fa616bb9be2e-kube-api-access-k65hx\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:47:31.640167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.640077 6208 generic.go:356] "Generic (PLEG): container finished" podID="c07df26b-e714-4555-a633-fa616bb9be2e" containerID="2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db" exitCode=0 Nov 04 00:47:31.640167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.640122 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" Nov 04 00:47:31.640689 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.640162 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" event={"ID":"c07df26b-e714-4555-a633-fa616bb9be2e","Type":"ContainerDied","Data":"2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db"} Nov 04 00:47:31.640689 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.640212 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-4dsdb" event={"ID":"c07df26b-e714-4555-a633-fa616bb9be2e","Type":"ContainerDied","Data":"fc97e6d5c45324b46dc8f35b6261164f07fd82dfad1157aaec3e0f91c7eaa2df"} Nov 04 00:47:31.640689 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.640233 6208 scope.go:122] "RemoveContainer" containerID="2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db" Nov 04 00:47:31.658474 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.658442 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-4dsdb"] Nov 04 00:47:31.661936 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.661914 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-4dsdb"] Nov 04 00:47:31.680910 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.680885 6208 scope.go:122] "RemoveContainer" containerID="2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db" Nov 04 00:47:31.681234 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:47:31.681216 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db\": container with ID starting with 2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db not found: ID does not exist" containerID="2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db" Nov 04 00:47:31.681308 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:31.681240 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db"} err="failed to get container status \"2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db\": rpc error: code = NotFound desc = could not find container \"2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db\": container with ID starting with 2e16e8fe4f9fb0a6342ecf9b451cdc7b39e49bfac7c5953a0b617e4b63abc4db not found: ID does not exist" Nov 04 00:47:33.293531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:33.293498 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07df26b-e714-4555-a633-fa616bb9be2e" path="/var/lib/kubelet/pods/c07df26b-e714-4555-a633-fa616bb9be2e/volumes" Nov 04 00:47:35.706153 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:47:35.706113 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:47:37.629110 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.629067 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b99c78677-pfh6c"] Nov 04 00:47:37.629576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.629451 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a0e5a1e6-cb54-4115-ae52-3278607679c5" containerName="payload-pre-processing" Nov 04 00:47:37.629576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.629464 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e5a1e6-cb54-4115-ae52-3278607679c5" containerName="payload-pre-processing" Nov 04 00:47:37.629576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.629502 6208 cpu_manager.go:414] "containerMap: removing container" podUID="c07df26b-e714-4555-a633-fa616bb9be2e" containerName="payload-processing" Nov 04 00:47:37.629576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.629507 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07df26b-e714-4555-a633-fa616bb9be2e" containerName="payload-processing" Nov 04 00:47:37.629576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.629564 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07df26b-e714-4555-a633-fa616bb9be2e" containerName="payload-processing" Nov 04 00:47:37.629576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.629573 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e5a1e6-cb54-4115-ae52-3278607679c5" containerName="payload-pre-processing" Nov 04 00:47:37.634278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.634261 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:37.642688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.642665 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b99c78677-pfh6c"] Nov 04 00:47:37.712917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.712887 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7ps\" (UniqueName: \"kubernetes.io/projected/7f1e58d6-90eb-44a5-a09d-2680a93e4e31-kube-api-access-sl7ps\") pod \"payload-processing-5b99c78677-pfh6c\" (UID: \"7f1e58d6-90eb-44a5-a09d-2680a93e4e31\") " pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:37.713103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.712925 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1e58d6-90eb-44a5-a09d-2680a93e4e31-plugins-config-volume\") pod \"payload-processing-5b99c78677-pfh6c\" (UID: \"7f1e58d6-90eb-44a5-a09d-2680a93e4e31\") " pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:37.788424 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.788387 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-668f485779-rptm9"] Nov 04 00:47:37.792212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.792193 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:37.801419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.801397 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-668f485779-rptm9"] Nov 04 00:47:37.813454 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.813430 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7ps\" (UniqueName: \"kubernetes.io/projected/7f1e58d6-90eb-44a5-a09d-2680a93e4e31-kube-api-access-sl7ps\") pod \"payload-processing-5b99c78677-pfh6c\" (UID: \"7f1e58d6-90eb-44a5-a09d-2680a93e4e31\") " pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:37.813586 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.813463 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1e58d6-90eb-44a5-a09d-2680a93e4e31-plugins-config-volume\") pod \"payload-processing-5b99c78677-pfh6c\" (UID: \"7f1e58d6-90eb-44a5-a09d-2680a93e4e31\") " pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:37.814101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.814083 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1e58d6-90eb-44a5-a09d-2680a93e4e31-plugins-config-volume\") pod \"payload-processing-5b99c78677-pfh6c\" (UID: \"7f1e58d6-90eb-44a5-a09d-2680a93e4e31\") " pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:37.827419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.827394 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7ps\" (UniqueName: \"kubernetes.io/projected/7f1e58d6-90eb-44a5-a09d-2680a93e4e31-kube-api-access-sl7ps\") pod \"payload-processing-5b99c78677-pfh6c\" (UID: \"7f1e58d6-90eb-44a5-a09d-2680a93e4e31\") " pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:37.914838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.914741 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f63c561-d450-4ad8-936a-3a035f80035c-plugins-config-volume\") pod \"payload-pre-processing-668f485779-rptm9\" (UID: \"3f63c561-d450-4ad8-936a-3a035f80035c\") " pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:37.914838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.914803 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqnl\" (UniqueName: \"kubernetes.io/projected/3f63c561-d450-4ad8-936a-3a035f80035c-kube-api-access-mpqnl\") pod \"payload-pre-processing-668f485779-rptm9\" (UID: \"3f63c561-d450-4ad8-936a-3a035f80035c\") " pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:37.949276 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:37.949250 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:38.015801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.015754 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f63c561-d450-4ad8-936a-3a035f80035c-plugins-config-volume\") pod \"payload-pre-processing-668f485779-rptm9\" (UID: \"3f63c561-d450-4ad8-936a-3a035f80035c\") " pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:38.015974 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.015846 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqnl\" (UniqueName: \"kubernetes.io/projected/3f63c561-d450-4ad8-936a-3a035f80035c-kube-api-access-mpqnl\") pod \"payload-pre-processing-668f485779-rptm9\" (UID: \"3f63c561-d450-4ad8-936a-3a035f80035c\") " pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:38.017213 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.017183 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f63c561-d450-4ad8-936a-3a035f80035c-plugins-config-volume\") pod \"payload-pre-processing-668f485779-rptm9\" (UID: \"3f63c561-d450-4ad8-936a-3a035f80035c\") " pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:38.032059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.029426 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqnl\" (UniqueName: \"kubernetes.io/projected/3f63c561-d450-4ad8-936a-3a035f80035c-kube-api-access-mpqnl\") pod \"payload-pre-processing-668f485779-rptm9\" (UID: \"3f63c561-d450-4ad8-936a-3a035f80035c\") " pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:38.106228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.106189 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:38.225912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.225887 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b99c78677-pfh6c"] Nov 04 00:47:38.228063 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:47:38.228025 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1e58d6_90eb_44a5_a09d_2680a93e4e31.slice/crio-c0bfac9b1e4dafe24a76a6143eb48e4cb2159395eda4730a04650687fdf4ce08 WatchSource:0}: Error finding container c0bfac9b1e4dafe24a76a6143eb48e4cb2159395eda4730a04650687fdf4ce08: Status 404 returned error can't find the container with id c0bfac9b1e4dafe24a76a6143eb48e4cb2159395eda4730a04650687fdf4ce08 Nov 04 00:47:38.386771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.386748 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-668f485779-rptm9"] Nov 04 00:47:38.388539 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:47:38.388507 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f63c561_d450_4ad8_936a_3a035f80035c.slice/crio-b4ec6cbb1b6d251ff4bf7a06f2413a3ad9554dad7dfc6b1358074dcc74e149b5 WatchSource:0}: Error finding container b4ec6cbb1b6d251ff4bf7a06f2413a3ad9554dad7dfc6b1358074dcc74e149b5: Status 404 returned error can't find the container with id b4ec6cbb1b6d251ff4bf7a06f2413a3ad9554dad7dfc6b1358074dcc74e149b5 Nov 04 00:47:38.680788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.680693 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" event={"ID":"3f63c561-d450-4ad8-936a-3a035f80035c","Type":"ContainerStarted","Data":"ab820acc6f120cca199585551340e0d4666a389a312b99663f5a94fced7c1d8d"} Nov 04 00:47:38.680788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.680750 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" event={"ID":"3f63c561-d450-4ad8-936a-3a035f80035c","Type":"ContainerStarted","Data":"b4ec6cbb1b6d251ff4bf7a06f2413a3ad9554dad7dfc6b1358074dcc74e149b5"} Nov 04 00:47:38.681411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.680875 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:38.682328 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.682302 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" event={"ID":"7f1e58d6-90eb-44a5-a09d-2680a93e4e31","Type":"ContainerStarted","Data":"834b0a6322de3d7e5fb945e956f2bbe0c3e611a842250b38e9bf90a0178867be"} Nov 04 00:47:38.682456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.682343 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" event={"ID":"7f1e58d6-90eb-44a5-a09d-2680a93e4e31","Type":"ContainerStarted","Data":"c0bfac9b1e4dafe24a76a6143eb48e4cb2159395eda4730a04650687fdf4ce08"} Nov 04 00:47:38.682456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.682434 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:38.703368 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.703325 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" podStartSLOduration=1.7033118470000002 podStartE2EDuration="1.703311847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:47:38.700101624 +0000 UTC m=+4677.522697788" watchObservedRunningTime="2026-11-04 00:47:38.703311847 +0000 UTC m=+4677.525908000" Nov 04 00:47:38.721336 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:38.721279 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" podStartSLOduration=1.72126808 podStartE2EDuration="1.72126808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:47:38.719183445 +0000 UTC m=+4677.541779610" watchObservedRunningTime="2026-11-04 00:47:38.72126808 +0000 UTC m=+4677.543864242" Nov 04 00:47:47.097358 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:47.097321 6208 scope.go:122] "RemoveContainer" containerID="68481c80428c84a9a5ee0eb8ecb3eedb7142bda29d4af5a42a05a82959563da6" Nov 04 00:47:47.106812 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:47.106785 6208 scope.go:122] "RemoveContainer" containerID="a073fa164ab5511051dfa68e46c94557b9cc3089c90a1b4027069ff50b45ffaf" Nov 04 00:47:49.691380 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:49.691291 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b99c78677-pfh6c" Nov 04 00:47:49.691829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:49.691517 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-668f485779-rptm9" Nov 04 00:47:49.746462 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:49.746425 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-684b8f58d7-s7fth"] Nov 04 00:47:49.772087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:49.772021 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws"] Nov 04 00:47:54.747657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:54.747592 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" podUID="a2430d0a-1990-4e53-8297-241b76baae8b" containerName="payload-processing" containerID="cri-o://d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8" gracePeriod=25 Nov 04 00:47:54.772335 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:54.772305 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" podUID="a7556abd-0a59-4d61-a928-9b9eb72f7e81" containerName="payload-pre-processing" containerID="cri-o://3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822" gracePeriod=25 Nov 04 00:47:55.020454 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.020426 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:55.056913 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.056884 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:47:55.187854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.187814 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2430d0a-1990-4e53-8297-241b76baae8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2430d0a-1990-4e53-8297-241b76baae8b-plugins-config-volume\") pod \"a2430d0a-1990-4e53-8297-241b76baae8b\" (UID: \"a2430d0a-1990-4e53-8297-241b76baae8b\") " Nov 04 00:47:55.188028 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.187876 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7556abd-0a59-4d61-a928-9b9eb72f7e81-kube-api-access-ccfpx\" (UniqueName: \"kubernetes.io/projected/a7556abd-0a59-4d61-a928-9b9eb72f7e81-kube-api-access-ccfpx\") pod \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\" (UID: \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\") " Nov 04 00:47:55.188189 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.188167 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2430d0a-1990-4e53-8297-241b76baae8b-kube-api-access-9r9bw\" (UniqueName: \"kubernetes.io/projected/a2430d0a-1990-4e53-8297-241b76baae8b-kube-api-access-9r9bw\") pod \"a2430d0a-1990-4e53-8297-241b76baae8b\" (UID: \"a2430d0a-1990-4e53-8297-241b76baae8b\") " Nov 04 00:47:55.188273 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.188249 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7556abd-0a59-4d61-a928-9b9eb72f7e81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7556abd-0a59-4d61-a928-9b9eb72f7e81-plugins-config-volume\") pod \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\" (UID: \"a7556abd-0a59-4d61-a928-9b9eb72f7e81\") " Nov 04 00:47:55.188507 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.188482 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2430d0a-1990-4e53-8297-241b76baae8b-plugins-config-volume" pod "a2430d0a-1990-4e53-8297-241b76baae8b" (UID: "a2430d0a-1990-4e53-8297-241b76baae8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:47:55.188748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.188716 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2430d0a-1990-4e53-8297-241b76baae8b-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:47:55.188935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.188913 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7556abd-0a59-4d61-a928-9b9eb72f7e81-plugins-config-volume" pod "a7556abd-0a59-4d61-a928-9b9eb72f7e81" (UID: "a7556abd-0a59-4d61-a928-9b9eb72f7e81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:47:55.190647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.190618 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7556abd-0a59-4d61-a928-9b9eb72f7e81-kube-api-access-ccfpx" pod "a7556abd-0a59-4d61-a928-9b9eb72f7e81" (UID: "a7556abd-0a59-4d61-a928-9b9eb72f7e81"). InnerVolumeSpecName "kube-api-access-ccfpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:47:55.190728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.190658 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2430d0a-1990-4e53-8297-241b76baae8b-kube-api-access-9r9bw" pod "a2430d0a-1990-4e53-8297-241b76baae8b" (UID: "a2430d0a-1990-4e53-8297-241b76baae8b"). InnerVolumeSpecName "kube-api-access-9r9bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:47:55.289979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.289893 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccfpx\" (UniqueName: \"kubernetes.io/projected/a7556abd-0a59-4d61-a928-9b9eb72f7e81-kube-api-access-ccfpx\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:47:55.289979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.289930 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r9bw\" (UniqueName: \"kubernetes.io/projected/a2430d0a-1990-4e53-8297-241b76baae8b-kube-api-access-9r9bw\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:47:55.289979 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.289957 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7556abd-0a59-4d61-a928-9b9eb72f7e81-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:47:55.779252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.779219 6208 generic.go:356] "Generic (PLEG): container finished" podID="a2430d0a-1990-4e53-8297-241b76baae8b" containerID="d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8" exitCode=0 Nov 04 00:47:55.779659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.779259 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" Nov 04 00:47:55.779659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.779303 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" event={"ID":"a2430d0a-1990-4e53-8297-241b76baae8b","Type":"ContainerDied","Data":"d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8"} Nov 04 00:47:55.779659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.779368 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-684b8f58d7-s7fth" event={"ID":"a2430d0a-1990-4e53-8297-241b76baae8b","Type":"ContainerDied","Data":"b4ef2e91915ee2c79ca731679b00890dbe925c7adea1db86a0e3746ba17e0a49"} Nov 04 00:47:55.779659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.779404 6208 scope.go:122] "RemoveContainer" containerID="d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8" Nov 04 00:47:55.780801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.780777 6208 generic.go:356] "Generic (PLEG): container finished" podID="a7556abd-0a59-4d61-a928-9b9eb72f7e81" containerID="3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822" exitCode=0 Nov 04 00:47:55.780881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.780826 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" Nov 04 00:47:55.780881 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.780835 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" event={"ID":"a7556abd-0a59-4d61-a928-9b9eb72f7e81","Type":"ContainerDied","Data":"3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822"} Nov 04 00:47:55.781009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.780886 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws" event={"ID":"a7556abd-0a59-4d61-a928-9b9eb72f7e81","Type":"ContainerDied","Data":"997ee1033c260df9e7da84da1afdb57b3ff83c5b4807cb80f5199112fd4c5ee0"} Nov 04 00:47:55.789297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.789279 6208 scope.go:122] "RemoveContainer" containerID="d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8" Nov 04 00:47:55.789543 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:47:55.789526 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8\": container with ID starting with d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8 not found: ID does not exist" containerID="d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8" Nov 04 00:47:55.789603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.789552 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8"} err="failed to get container status \"d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8\": rpc error: code = NotFound desc = could not find container \"d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8\": container with ID starting with d90a5f82c81b383d7c37b14ca19bc83adbf31734f4743badc307a273935278f8 not found: ID does not exist" Nov 04 00:47:55.789603 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.789572 6208 scope.go:122] "RemoveContainer" containerID="3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822" Nov 04 00:47:55.797424 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.797401 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-684b8f58d7-s7fth"] Nov 04 00:47:55.797741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.797710 6208 scope.go:122] "RemoveContainer" containerID="3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822" Nov 04 00:47:55.797995 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:47:55.797979 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822\": container with ID starting with 3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822 not found: ID does not exist" containerID="3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822" Nov 04 00:47:55.798079 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.798001 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822"} err="failed to get container status \"3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822\": rpc error: code = NotFound desc = could not find container \"3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822\": container with ID starting with 3215364df4d69ac6b9fb3e97cb280cd926024068cd86c55e4db99baf18dcd822 not found: ID does not exist" Nov 04 00:47:55.800733 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.800711 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-684b8f58d7-s7fth"] Nov 04 00:47:55.812221 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.812198 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws"] Nov 04 00:47:55.816954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:55.816930 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-5cb47fbf9b-lhcws"] Nov 04 00:47:57.292727 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:57.292690 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2430d0a-1990-4e53-8297-241b76baae8b" path="/var/lib/kubelet/pods/a2430d0a-1990-4e53-8297-241b76baae8b/volumes" Nov 04 00:47:57.293148 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:57.293087 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7556abd-0a59-4d61-a928-9b9eb72f7e81" path="/var/lib/kubelet/pods/a7556abd-0a59-4d61-a928-9b9eb72f7e81/volumes" Nov 04 00:47:58.647767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:47:58.647730 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:02.019754 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.019718 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:02.196672 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.196639 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:02.529406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.529372 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:02.554843 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.554802 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr"] Nov 04 00:48:02.555202 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.555188 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a2430d0a-1990-4e53-8297-241b76baae8b" containerName="payload-processing" Nov 04 00:48:02.555260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.555203 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2430d0a-1990-4e53-8297-241b76baae8b" containerName="payload-processing" Nov 04 00:48:02.555260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.555215 6208 cpu_manager.go:414] "containerMap: removing container" podUID="a7556abd-0a59-4d61-a928-9b9eb72f7e81" containerName="payload-pre-processing" Nov 04 00:48:02.555260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.555220 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7556abd-0a59-4d61-a928-9b9eb72f7e81" containerName="payload-pre-processing" Nov 04 00:48:02.555371 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.555296 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2430d0a-1990-4e53-8297-241b76baae8b" containerName="payload-processing" Nov 04 00:48:02.555371 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.555305 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7556abd-0a59-4d61-a928-9b9eb72f7e81" containerName="payload-pre-processing" Nov 04 00:48:02.559885 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.559864 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.563126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.563105 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.563256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.563130 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.563256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.563160 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976cj\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-kube-api-access-976cj\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.563256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.563179 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.563977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.563958 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 04 00:48:02.567384 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.567328 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 04 00:48:02.567481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.567386 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 04 00:48:02.568836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.568812 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 04 00:48:02.571188 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.571163 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr"] Nov 04 00:48:02.664184 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.664145 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 04 00:48:02.664536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.664510 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.664634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.664545 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.664634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.664590 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976cj\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-kube-api-access-976cj\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.664634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.664615 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.665298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.665275 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.665736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.665455 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.670920 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.667855 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.677665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.677629 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976cj\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-kube-api-access-976cj\") pod \"payload-processing-external-model-7fd6967f4c-w27zr\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:02.874752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:02.874664 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:03.153203 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:03.153174 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr"] Nov 04 00:48:03.155501 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:48:03.155471 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408bcaf9_a98a_4d15_956a_003ef919106a.slice/crio-c991e41f5411c2056de6e6673e33647e9cf34603338057da0e2fe71b95f5962a WatchSource:0}: Error finding container c991e41f5411c2056de6e6673e33647e9cf34603338057da0e2fe71b95f5962a: Status 404 returned error can't find the container with id c991e41f5411c2056de6e6673e33647e9cf34603338057da0e2fe71b95f5962a Nov 04 00:48:03.825507 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:03.825459 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" event={"ID":"408bcaf9-a98a-4d15-956a-003ef919106a","Type":"ContainerStarted","Data":"9742e3ad3280883506f9276f9fba9d5903771442559bbf6e7ea5d4ba32e10880"} Nov 04 00:48:03.826058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:03.825525 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" event={"ID":"408bcaf9-a98a-4d15-956a-003ef919106a","Type":"ContainerStarted","Data":"c991e41f5411c2056de6e6673e33647e9cf34603338057da0e2fe71b95f5962a"} Nov 04 00:48:03.826058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:03.825654 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:03.844885 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:03.844833 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" podStartSLOduration=1.844820305 podStartE2EDuration="1.844820305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:48:03.843842901 +0000 UTC m=+4702.666439064" watchObservedRunningTime="2026-11-04 00:48:03.844820305 +0000 UTC m=+4702.667416466" Nov 04 00:48:05.188896 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:05.188827 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:05.772677 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:48:05.772621 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:48:07.923842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:07.923803 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:08.026083 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:08.026051 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:14.836606 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:14.836573 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:48:24.608240 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:24.608193 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:35.704438 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:48:35.704397 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:48:54.156330 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:54.156287 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:55.607150 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:55.607113 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:48:58.706619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:48:58.706579 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:49:05.702343 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:49:05.702302 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:49:10.026841 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:10.026800 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l"] Nov 04 00:49:10.029282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:10.029255 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a948a352-ljt7l"] Nov 04 00:49:11.292491 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:11.292455 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67a52df-2c6c-4eba-877f-e9a87e3d18e1" path="/var/lib/kubelet/pods/a67a52df-2c6c-4eba-877f-e9a87e3d18e1/volumes" Nov 04 00:49:15.274858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:15.274820 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:49:24.657492 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:24.657406 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:49:29.520912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.520875 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k"] Nov 04 00:49:29.524466 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.524447 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.525079 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.525021 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97447\" (UniqueName: \"kubernetes.io/projected/89202ebf-5bce-4078-ad92-9bd176d929f7-kube-api-access-97447\") pod \"maas-api-revoke-sub-e2547622-dhh5k\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.525223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.525163 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-dhh5k\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.525223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.525201 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-dhh5k\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.528542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.528520 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:49:29.530003 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.529971 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:49:29.530195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.530171 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:49:29.530303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.530254 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:49:29.535199 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.535179 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k"] Nov 04 00:49:29.625554 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.625509 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97447\" (UniqueName: \"kubernetes.io/projected/89202ebf-5bce-4078-ad92-9bd176d929f7-kube-api-access-97447\") pod \"maas-api-revoke-sub-e2547622-dhh5k\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.625768 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.625576 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-dhh5k\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.625768 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.625598 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-dhh5k\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.626278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.626252 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-dhh5k\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.628218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.628195 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-dhh5k\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.638122 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.638099 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97447\" (UniqueName: \"kubernetes.io/projected/89202ebf-5bce-4078-ad92-9bd176d929f7-kube-api-access-97447\") pod \"maas-api-revoke-sub-e2547622-dhh5k\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:29.839100 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:29.839012 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:30.115297 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:30.115217 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k"] Nov 04 00:49:30.118951 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:49:30.118922 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89202ebf_5bce_4078_ad92_9bd176d929f7.slice/crio-1312e944feda9a12367a39b723e767cb3814c6ac19da517452e32de36b635f7e WatchSource:0}: Error finding container 1312e944feda9a12367a39b723e767cb3814c6ac19da517452e32de36b635f7e: Status 404 returned error can't find the container with id 1312e944feda9a12367a39b723e767cb3814c6ac19da517452e32de36b635f7e Nov 04 00:49:30.281385 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:30.281349 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" event={"ID":"89202ebf-5bce-4078-ad92-9bd176d929f7","Type":"ContainerStarted","Data":"1312e944feda9a12367a39b723e767cb3814c6ac19da517452e32de36b635f7e"} Nov 04 00:49:31.288723 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:31.288686 6208 generic.go:356] "Generic (PLEG): container finished" podID="89202ebf-5bce-4078-ad92-9bd176d929f7" containerID="ce5163dd655a283603cf4419ec97f63f333fdff747e95b434b2f8feed97011ed" exitCode=0 Nov 04 00:49:31.292532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:31.292502 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" event={"ID":"89202ebf-5bce-4078-ad92-9bd176d929f7","Type":"ContainerDied","Data":"ce5163dd655a283603cf4419ec97f63f333fdff747e95b434b2f8feed97011ed"} Nov 04 00:49:32.425829 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.425802 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:32.451509 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.451479 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89202ebf-5bce-4078-ad92-9bd176d929f7-kube-api-access-97447\" (UniqueName: \"kubernetes.io/projected/89202ebf-5bce-4078-ad92-9bd176d929f7-kube-api-access-97447\") pod \"89202ebf-5bce-4078-ad92-9bd176d929f7\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " Nov 04 00:49:32.451697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.451551 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-xks-ca\") pod \"89202ebf-5bce-4078-ad92-9bd176d929f7\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " Nov 04 00:49:32.451697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.451626 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-openshift-ca\") pod \"89202ebf-5bce-4078-ad92-9bd176d929f7\" (UID: \"89202ebf-5bce-4078-ad92-9bd176d929f7\") " Nov 04 00:49:32.452246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.452213 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-openshift-ca" pod "89202ebf-5bce-4078-ad92-9bd176d929f7" (UID: "89202ebf-5bce-4078-ad92-9bd176d929f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:49:32.454953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.454923 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-xks-ca" pod "89202ebf-5bce-4078-ad92-9bd176d929f7" (UID: "89202ebf-5bce-4078-ad92-9bd176d929f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:49:32.455219 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.455195 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89202ebf-5bce-4078-ad92-9bd176d929f7-kube-api-access-97447" pod "89202ebf-5bce-4078-ad92-9bd176d929f7" (UID: "89202ebf-5bce-4078-ad92-9bd176d929f7"). InnerVolumeSpecName "kube-api-access-97447". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:49:32.552822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.552731 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:32.552822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.552767 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97447\" (UniqueName: \"kubernetes.io/projected/89202ebf-5bce-4078-ad92-9bd176d929f7-kube-api-access-97447\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:32.552822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:32.552777 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89202ebf-5bce-4078-ad92-9bd176d929f7-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:33.300662 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:33.300634 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" Nov 04 00:49:33.300835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:33.300670 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k" event={"ID":"89202ebf-5bce-4078-ad92-9bd176d929f7","Type":"ContainerDied","Data":"1312e944feda9a12367a39b723e767cb3814c6ac19da517452e32de36b635f7e"} Nov 04 00:49:33.300835 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:33.300715 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1312e944feda9a12367a39b723e767cb3814c6ac19da517452e32de36b635f7e" Nov 04 00:49:35.704073 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:49:35.704015 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:49:40.227825 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:40.227782 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr"] Nov 04 00:49:40.263911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:40.263876 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:49:40.372197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:40.372145 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:49:40.550419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:40.550316 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:49:43.148848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:43.148813 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:49:43.151976 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:43.151952 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:49:45.229346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.229094 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" podUID="408bcaf9-a98a-4d15-956a-003ef919106a" containerName="payload-processing" containerID="cri-o://9742e3ad3280883506f9276f9fba9d5903771442559bbf6e7ea5d4ba32e10880" gracePeriod=25 Nov 04 00:49:45.429665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.429611 6208 generic.go:356] "Generic (PLEG): container finished" podID="408bcaf9-a98a-4d15-956a-003ef919106a" containerID="9742e3ad3280883506f9276f9fba9d5903771442559bbf6e7ea5d4ba32e10880" exitCode=0 Nov 04 00:49:45.429977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.429653 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" event={"ID":"408bcaf9-a98a-4d15-956a-003ef919106a","Type":"ContainerDied","Data":"9742e3ad3280883506f9276f9fba9d5903771442559bbf6e7ea5d4ba32e10880"} Nov 04 00:49:45.656992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.656961 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:49:45.841503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.841399 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-kube-api-access-976cj\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-kube-api-access-976cj\") pod \"408bcaf9-a98a-4d15-956a-003ef919106a\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " Nov 04 00:49:45.841503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.841466 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-plugins-config-volume\") pod \"408bcaf9-a98a-4d15-956a-003ef919106a\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " Nov 04 00:49:45.841764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.841535 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-routing-overlay-volume\") pod \"408bcaf9-a98a-4d15-956a-003ef919106a\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " Nov 04 00:49:45.841764 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.841583 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-provider-credentials\") pod \"408bcaf9-a98a-4d15-956a-003ef919106a\" (UID: \"408bcaf9-a98a-4d15-956a-003ef919106a\") " Nov 04 00:49:45.841953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.841920 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-plugins-config-volume" pod "408bcaf9-a98a-4d15-956a-003ef919106a" (UID: "408bcaf9-a98a-4d15-956a-003ef919106a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:49:45.842059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.841956 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-routing-overlay-volume" pod "408bcaf9-a98a-4d15-956a-003ef919106a" (UID: "408bcaf9-a98a-4d15-956a-003ef919106a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:49:45.844057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.844008 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-provider-credentials" pod "408bcaf9-a98a-4d15-956a-003ef919106a" (UID: "408bcaf9-a98a-4d15-956a-003ef919106a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:49:45.844372 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.844355 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-kube-api-access-976cj" pod "408bcaf9-a98a-4d15-956a-003ef919106a" (UID: "408bcaf9-a98a-4d15-956a-003ef919106a"). InnerVolumeSpecName "kube-api-access-976cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:49:45.942821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.942782 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-976cj\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-kube-api-access-976cj\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:45.942821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.942817 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:45.943046 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.942830 6208 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/408bcaf9-a98a-4d15-956a-003ef919106a-routing-overlay-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:45.943046 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:45.942846 6208 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/408bcaf9-a98a-4d15-956a-003ef919106a-provider-credentials\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:46.437900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:46.437858 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" Nov 04 00:49:46.438366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:46.437861 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr" event={"ID":"408bcaf9-a98a-4d15-956a-003ef919106a","Type":"ContainerDied","Data":"c991e41f5411c2056de6e6673e33647e9cf34603338057da0e2fe71b95f5962a"} Nov 04 00:49:46.438366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:46.438050 6208 scope.go:122] "RemoveContainer" containerID="9742e3ad3280883506f9276f9fba9d5903771442559bbf6e7ea5d4ba32e10880" Nov 04 00:49:46.471648 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:46.471608 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr"] Nov 04 00:49:46.471800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:46.471671 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-w27zr"] Nov 04 00:49:47.316058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:47.306978 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408bcaf9-a98a-4d15-956a-003ef919106a" path="/var/lib/kubelet/pods/408bcaf9-a98a-4d15-956a-003ef919106a/volumes" Nov 04 00:49:47.544252 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:47.544205 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:49:47.544883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:47.544844 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:49:48.199227 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:48.199180 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:49:48.199427 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:48.199269 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:49:49.891978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.891936 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c"] Nov 04 00:49:49.892418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.892354 6208 cpu_manager.go:414] "containerMap: removing container" podUID="89202ebf-5bce-4078-ad92-9bd176d929f7" containerName="revoke-keys" Nov 04 00:49:49.892418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.892371 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89202ebf-5bce-4078-ad92-9bd176d929f7" containerName="revoke-keys" Nov 04 00:49:49.892418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.892384 6208 cpu_manager.go:414] "containerMap: removing container" podUID="408bcaf9-a98a-4d15-956a-003ef919106a" containerName="payload-processing" Nov 04 00:49:49.892418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.892390 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408bcaf9-a98a-4d15-956a-003ef919106a" containerName="payload-processing" Nov 04 00:49:49.892561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.892466 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="89202ebf-5bce-4078-ad92-9bd176d929f7" containerName="revoke-keys" Nov 04 00:49:49.892561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.892475 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="408bcaf9-a98a-4d15-956a-003ef919106a" containerName="payload-processing" Nov 04 00:49:49.897290 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.897261 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:49.904692 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.904665 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:49:49.904692 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.904665 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:49:49.904945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.904746 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:49:49.904945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.904833 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:49:49.908101 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:49.908076 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c"] Nov 04 00:49:50.020772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.020732 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnp5c\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.020978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.020880 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnp5c\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.020978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.020916 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/016f274c-500d-4ccb-9749-205f1b4c1f48-kube-api-access-nn4zn\") pod \"maas-api-revoke-sub-c1e271a9-pnp5c\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.121844 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.121802 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnp5c\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.122029 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.121885 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnp5c\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.122029 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.121907 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/016f274c-500d-4ccb-9749-205f1b4c1f48-kube-api-access-nn4zn\") pod \"maas-api-revoke-sub-c1e271a9-pnp5c\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.122639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.122613 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnp5c\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.124363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.124340 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnp5c\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.134247 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.134225 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/016f274c-500d-4ccb-9749-205f1b4c1f48-kube-api-access-nn4zn\") pod \"maas-api-revoke-sub-c1e271a9-pnp5c\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.213430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.213397 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:50.488667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:50.488611 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c"] Nov 04 00:49:50.491159 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:49:50.491113 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016f274c_500d_4ccb_9749_205f1b4c1f48.slice/crio-5ad79abe237661f3ee5a25ac7cc58764b51c70c03b67c9b674ba613a51fb15b7 WatchSource:0}: Error finding container 5ad79abe237661f3ee5a25ac7cc58764b51c70c03b67c9b674ba613a51fb15b7: Status 404 returned error can't find the container with id 5ad79abe237661f3ee5a25ac7cc58764b51c70c03b67c9b674ba613a51fb15b7 Nov 04 00:49:51.468853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:51.468813 6208 generic.go:356] "Generic (PLEG): container finished" podID="016f274c-500d-4ccb-9749-205f1b4c1f48" containerID="970262f51e940375bfab4e09dd478427f2b59a7e74d5d40fd0ea2be499eab00d" exitCode=0 Nov 04 00:49:51.469270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:51.468900 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" event={"ID":"016f274c-500d-4ccb-9749-205f1b4c1f48","Type":"ContainerDied","Data":"970262f51e940375bfab4e09dd478427f2b59a7e74d5d40fd0ea2be499eab00d"} Nov 04 00:49:51.469270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:51.468952 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" event={"ID":"016f274c-500d-4ccb-9749-205f1b4c1f48","Type":"ContainerStarted","Data":"5ad79abe237661f3ee5a25ac7cc58764b51c70c03b67c9b674ba613a51fb15b7"} Nov 04 00:49:52.608365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.608338 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:52.749507 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.749423 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-xks-ca\") pod \"016f274c-500d-4ccb-9749-205f1b4c1f48\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " Nov 04 00:49:52.749507 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.749498 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016f274c-500d-4ccb-9749-205f1b4c1f48-kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/016f274c-500d-4ccb-9749-205f1b4c1f48-kube-api-access-nn4zn\") pod \"016f274c-500d-4ccb-9749-205f1b4c1f48\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " Nov 04 00:49:52.749709 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.749524 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-openshift-ca\") pod \"016f274c-500d-4ccb-9749-205f1b4c1f48\" (UID: \"016f274c-500d-4ccb-9749-205f1b4c1f48\") " Nov 04 00:49:52.749935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.749910 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-openshift-ca" pod "016f274c-500d-4ccb-9749-205f1b4c1f48" (UID: "016f274c-500d-4ccb-9749-205f1b4c1f48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:49:52.751520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.751491 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-xks-ca" pod "016f274c-500d-4ccb-9749-205f1b4c1f48" (UID: "016f274c-500d-4ccb-9749-205f1b4c1f48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:49:52.751644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.751621 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016f274c-500d-4ccb-9749-205f1b4c1f48-kube-api-access-nn4zn" pod "016f274c-500d-4ccb-9749-205f1b4c1f48" (UID: "016f274c-500d-4ccb-9749-205f1b4c1f48"). InnerVolumeSpecName "kube-api-access-nn4zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:49:52.851220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.851188 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:52.851220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.851215 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn4zn\" (UniqueName: \"kubernetes.io/projected/016f274c-500d-4ccb-9749-205f1b4c1f48-kube-api-access-nn4zn\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:52.851220 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:52.851228 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016f274c-500d-4ccb-9749-205f1b4c1f48-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:49:53.480503 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:53.480468 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" event={"ID":"016f274c-500d-4ccb-9749-205f1b4c1f48","Type":"ContainerDied","Data":"5ad79abe237661f3ee5a25ac7cc58764b51c70c03b67c9b674ba613a51fb15b7"} Nov 04 00:49:53.480668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:53.480520 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ad79abe237661f3ee5a25ac7cc58764b51c70c03b67c9b674ba613a51fb15b7" Nov 04 00:49:53.480668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:53.480484 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c" Nov 04 00:49:57.073508 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:57.073471 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:49:58.771346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:49:58.771300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:50:05.704839 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:50:05.704799 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:50:06.809935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.809902 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2"] Nov 04 00:50:06.810405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.810386 6208 cpu_manager.go:414] "containerMap: removing container" podUID="016f274c-500d-4ccb-9749-205f1b4c1f48" containerName="revoke-keys" Nov 04 00:50:06.810405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.810404 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016f274c-500d-4ccb-9749-205f1b4c1f48" containerName="revoke-keys" Nov 04 00:50:06.810520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.810482 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="016f274c-500d-4ccb-9749-205f1b4c1f48" containerName="revoke-keys" Nov 04 00:50:06.813828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.813811 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:06.852333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.852303 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2"] Nov 04 00:50:06.857389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.857363 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:50:06.857610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.857597 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:50:06.857694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.857626 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:50:06.857694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.857629 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:50:06.911476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.911448 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b6hp2\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:06.911630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.911484 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b6hp2\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:06.911630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:06.911504 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h6tj\" (UniqueName: \"kubernetes.io/projected/1f44bc50-0641-41f7-9288-f930dd6b3ca6-kube-api-access-8h6tj\") pod \"maas-api-revoke-sub-2e1f5b5f-b6hp2\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:07.012666 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:07.012637 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b6hp2\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:07.012805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:07.012672 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b6hp2\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:07.012805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:07.012693 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h6tj\" (UniqueName: \"kubernetes.io/projected/1f44bc50-0641-41f7-9288-f930dd6b3ca6-kube-api-access-8h6tj\") pod \"maas-api-revoke-sub-2e1f5b5f-b6hp2\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:07.013574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:07.013545 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b6hp2\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:07.015175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:07.015150 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-b6hp2\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:07.025433 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:07.025404 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h6tj\" (UniqueName: \"kubernetes.io/projected/1f44bc50-0641-41f7-9288-f930dd6b3ca6-kube-api-access-8h6tj\") pod \"maas-api-revoke-sub-2e1f5b5f-b6hp2\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:07.127076 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:07.126986 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:07.405413 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:07.405385 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2"] Nov 04 00:50:07.407562 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:50:07.407532 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f44bc50_0641_41f7_9288_f930dd6b3ca6.slice/crio-7bcfcfa09a92753f0dc9abcdf77a52a54abc60f2528184eeff31279244f2b0ed WatchSource:0}: Error finding container 7bcfcfa09a92753f0dc9abcdf77a52a54abc60f2528184eeff31279244f2b0ed: Status 404 returned error can't find the container with id 7bcfcfa09a92753f0dc9abcdf77a52a54abc60f2528184eeff31279244f2b0ed Nov 04 00:50:07.555865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:07.555823 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" event={"ID":"1f44bc50-0641-41f7-9288-f930dd6b3ca6","Type":"ContainerStarted","Data":"7bcfcfa09a92753f0dc9abcdf77a52a54abc60f2528184eeff31279244f2b0ed"} Nov 04 00:50:08.562756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:08.562719 6208 generic.go:356] "Generic (PLEG): container finished" podID="1f44bc50-0641-41f7-9288-f930dd6b3ca6" containerID="4d3343c4f040e003c262047ee471936a32e21ed182149ea6df7da39f2e06abde" exitCode=0 Nov 04 00:50:08.563171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:08.562805 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" event={"ID":"1f44bc50-0641-41f7-9288-f930dd6b3ca6","Type":"ContainerDied","Data":"4d3343c4f040e003c262047ee471936a32e21ed182149ea6df7da39f2e06abde"} Nov 04 00:50:09.699730 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.699697 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:09.843270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.843180 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-xks-ca\") pod \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " Nov 04 00:50:09.843270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.843246 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f44bc50-0641-41f7-9288-f930dd6b3ca6-kube-api-access-8h6tj\" (UniqueName: \"kubernetes.io/projected/1f44bc50-0641-41f7-9288-f930dd6b3ca6-kube-api-access-8h6tj\") pod \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " Nov 04 00:50:09.843270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.843272 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-openshift-ca\") pod \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\" (UID: \"1f44bc50-0641-41f7-9288-f930dd6b3ca6\") " Nov 04 00:50:09.843647 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.843623 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-openshift-ca" pod "1f44bc50-0641-41f7-9288-f930dd6b3ca6" (UID: "1f44bc50-0641-41f7-9288-f930dd6b3ca6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:50:09.845399 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.845367 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-xks-ca" pod "1f44bc50-0641-41f7-9288-f930dd6b3ca6" (UID: "1f44bc50-0641-41f7-9288-f930dd6b3ca6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:50:09.845539 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.845519 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f44bc50-0641-41f7-9288-f930dd6b3ca6-kube-api-access-8h6tj" pod "1f44bc50-0641-41f7-9288-f930dd6b3ca6" (UID: "1f44bc50-0641-41f7-9288-f930dd6b3ca6"). InnerVolumeSpecName "kube-api-access-8h6tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:50:09.944475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.944437 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:09.944475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.944472 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h6tj\" (UniqueName: \"kubernetes.io/projected/1f44bc50-0641-41f7-9288-f930dd6b3ca6-kube-api-access-8h6tj\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:09.944668 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:09.944487 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f44bc50-0641-41f7-9288-f930dd6b3ca6-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:10.573961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:10.573930 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" Nov 04 00:50:10.574156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:10.573965 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2" event={"ID":"1f44bc50-0641-41f7-9288-f930dd6b3ca6","Type":"ContainerDied","Data":"7bcfcfa09a92753f0dc9abcdf77a52a54abc60f2528184eeff31279244f2b0ed"} Nov 04 00:50:10.574156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:10.574017 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bcfcfa09a92753f0dc9abcdf77a52a54abc60f2528184eeff31279244f2b0ed" Nov 04 00:50:17.735999 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.735965 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p"] Nov 04 00:50:17.736615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.736476 6208 cpu_manager.go:414] "containerMap: removing container" podUID="1f44bc50-0641-41f7-9288-f930dd6b3ca6" containerName="revoke-keys" Nov 04 00:50:17.736615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.736494 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f44bc50-0641-41f7-9288-f930dd6b3ca6" containerName="revoke-keys" Nov 04 00:50:17.736615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.736596 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f44bc50-0641-41f7-9288-f930dd6b3ca6" containerName="revoke-keys" Nov 04 00:50:17.740064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.740042 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:17.747499 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.747473 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:50:17.747657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.747485 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:50:17.747842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.747527 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:50:17.747842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.747595 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:50:17.750205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.750181 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p"] Nov 04 00:50:17.924439 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.924394 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jbr4p\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:17.924616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.924458 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jbr4p\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:17.924616 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:17.924571 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzqp5\" (UniqueName: \"kubernetes.io/projected/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-kube-api-access-mzqp5\") pod \"maas-api-revoke-sub-ec843018-jbr4p\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:18.025745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:18.025659 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzqp5\" (UniqueName: \"kubernetes.io/projected/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-kube-api-access-mzqp5\") pod \"maas-api-revoke-sub-ec843018-jbr4p\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:18.025745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:18.025745 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jbr4p\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:18.025986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:18.025785 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jbr4p\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:18.026495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:18.026466 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jbr4p\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:18.027984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:18.027963 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jbr4p\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:18.038605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:18.038582 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzqp5\" (UniqueName: \"kubernetes.io/projected/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-kube-api-access-mzqp5\") pod \"maas-api-revoke-sub-ec843018-jbr4p\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:18.054520 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:18.054478 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:18.338025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:18.337999 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p"] Nov 04 00:50:18.340903 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:50:18.340869 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c94aa54_0479_48f3_b6a9_8cb4fe47150c.slice/crio-6c6e6de075a40c829a07ca499defa9750512490f2d312d212526e0e8951bd3ea WatchSource:0}: Error finding container 6c6e6de075a40c829a07ca499defa9750512490f2d312d212526e0e8951bd3ea: Status 404 returned error can't find the container with id 6c6e6de075a40c829a07ca499defa9750512490f2d312d212526e0e8951bd3ea Nov 04 00:50:18.619145 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:18.619059 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" event={"ID":"4c94aa54-0479-48f3-b6a9-8cb4fe47150c","Type":"ContainerStarted","Data":"6c6e6de075a40c829a07ca499defa9750512490f2d312d212526e0e8951bd3ea"} Nov 04 00:50:19.626697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:19.626663 6208 generic.go:356] "Generic (PLEG): container finished" podID="4c94aa54-0479-48f3-b6a9-8cb4fe47150c" containerID="ad53373fec4405ffa5a5e779fa9e1260f22d7f8fcfd7fcaa79ea7e2059e552e7" exitCode=0 Nov 04 00:50:19.627167 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:19.626726 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" event={"ID":"4c94aa54-0479-48f3-b6a9-8cb4fe47150c","Type":"ContainerDied","Data":"ad53373fec4405ffa5a5e779fa9e1260f22d7f8fcfd7fcaa79ea7e2059e552e7"} Nov 04 00:50:20.921809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:20.921433 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:21.077057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.070419 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-kube-api-access-mzqp5\" (UniqueName: \"kubernetes.io/projected/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-kube-api-access-mzqp5\") pod \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " Nov 04 00:50:21.077057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.070482 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-openshift-ca\") pod \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " Nov 04 00:50:21.077057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.070580 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-xks-ca\") pod \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\" (UID: \"4c94aa54-0479-48f3-b6a9-8cb4fe47150c\") " Nov 04 00:50:21.077057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.073373 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-openshift-ca" pod "4c94aa54-0479-48f3-b6a9-8cb4fe47150c" (UID: "4c94aa54-0479-48f3-b6a9-8cb4fe47150c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:50:21.102324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.098056 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-kube-api-access-mzqp5" pod "4c94aa54-0479-48f3-b6a9-8cb4fe47150c" (UID: "4c94aa54-0479-48f3-b6a9-8cb4fe47150c"). InnerVolumeSpecName "kube-api-access-mzqp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:50:21.122205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.122160 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-xks-ca" pod "4c94aa54-0479-48f3-b6a9-8cb4fe47150c" (UID: "4c94aa54-0479-48f3-b6a9-8cb4fe47150c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:50:21.186831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.186680 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:21.186831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.186730 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzqp5\" (UniqueName: \"kubernetes.io/projected/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-kube-api-access-mzqp5\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:21.186831 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.186747 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c94aa54-0479-48f3-b6a9-8cb4fe47150c-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:21.642201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.642167 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" Nov 04 00:50:21.642387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.642169 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p" event={"ID":"4c94aa54-0479-48f3-b6a9-8cb4fe47150c","Type":"ContainerDied","Data":"6c6e6de075a40c829a07ca499defa9750512490f2d312d212526e0e8951bd3ea"} Nov 04 00:50:21.642387 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:21.642290 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c6e6de075a40c829a07ca499defa9750512490f2d312d212526e0e8951bd3ea" Nov 04 00:50:24.721457 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:24.721416 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:50:33.202058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:33.200148 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:50:33.202058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:33.200273 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:50:35.703768 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:50:35.703725 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:50:44.388496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.388451 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf"] Nov 04 00:50:44.389134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.388971 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4c94aa54-0479-48f3-b6a9-8cb4fe47150c" containerName="revoke-keys" Nov 04 00:50:44.389134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.388993 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c94aa54-0479-48f3-b6a9-8cb4fe47150c" containerName="revoke-keys" Nov 04 00:50:44.389134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.389130 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c94aa54-0479-48f3-b6a9-8cb4fe47150c" containerName="revoke-keys" Nov 04 00:50:44.392318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.392297 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.400686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.400370 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:50:44.400686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.400454 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:50:44.400686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.400559 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:50:44.400686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.400574 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:50:44.405667 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.405633 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf"] Nov 04 00:50:44.450262 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.450227 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52tj2\" (UniqueName: \"kubernetes.io/projected/5c402cae-28ff-4bfc-a791-c62fc095ee20-kube-api-access-52tj2\") pod \"maas-api-revoke-sub-046e1343-x9whf\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.450429 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.450290 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-x9whf\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.450514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.450483 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-x9whf\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.551303 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.551264 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52tj2\" (UniqueName: \"kubernetes.io/projected/5c402cae-28ff-4bfc-a791-c62fc095ee20-kube-api-access-52tj2\") pod \"maas-api-revoke-sub-046e1343-x9whf\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.551482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.551336 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-x9whf\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.551482 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.551389 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-x9whf\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.552018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.551979 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-x9whf\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.553707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.553684 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-x9whf\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.563663 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.563639 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52tj2\" (UniqueName: \"kubernetes.io/projected/5c402cae-28ff-4bfc-a791-c62fc095ee20-kube-api-access-52tj2\") pod \"maas-api-revoke-sub-046e1343-x9whf\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.707914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.707887 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:44.987875 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:44.987846 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf"] Nov 04 00:50:44.990267 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:50:44.990238 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c402cae_28ff_4bfc_a791_c62fc095ee20.slice/crio-585a8e230d6bfe0b215d89aa3245f17986c705547c0be2c0f88056a77ff7fe84 WatchSource:0}: Error finding container 585a8e230d6bfe0b215d89aa3245f17986c705547c0be2c0f88056a77ff7fe84: Status 404 returned error can't find the container with id 585a8e230d6bfe0b215d89aa3245f17986c705547c0be2c0f88056a77ff7fe84 Nov 04 00:50:45.792374 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:45.792341 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" event={"ID":"5c402cae-28ff-4bfc-a791-c62fc095ee20","Type":"ContainerStarted","Data":"1243877ef76fab2976f9a33530805269f07d179abbf4a0a92d495ac1f3d2fdfe"} Nov 04 00:50:45.792755 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:45.792394 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" event={"ID":"5c402cae-28ff-4bfc-a791-c62fc095ee20","Type":"ContainerStarted","Data":"585a8e230d6bfe0b215d89aa3245f17986c705547c0be2c0f88056a77ff7fe84"} Nov 04 00:50:45.813461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:45.813411 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" podStartSLOduration=1.81339752 podStartE2EDuration="1.81339752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:50:45.8100863 +0000 UTC m=+4864.632682464" watchObservedRunningTime="2026-11-04 00:50:45.81339752 +0000 UTC m=+4864.635993682" Nov 04 00:50:46.799143 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:46.799108 6208 generic.go:356] "Generic (PLEG): container finished" podID="5c402cae-28ff-4bfc-a791-c62fc095ee20" containerID="1243877ef76fab2976f9a33530805269f07d179abbf4a0a92d495ac1f3d2fdfe" exitCode=0 Nov 04 00:50:46.799544 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:46.799189 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" event={"ID":"5c402cae-28ff-4bfc-a791-c62fc095ee20","Type":"ContainerDied","Data":"1243877ef76fab2976f9a33530805269f07d179abbf4a0a92d495ac1f3d2fdfe"} Nov 04 00:50:47.211357 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:47.211324 6208 scope.go:122] "RemoveContainer" containerID="6061c3ef475b9c16d8e5a80e597b9c4fdd33a91b60584446f8e10eae236380f7" Nov 04 00:50:47.944438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:47.944412 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:47.982998 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:47.982961 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-openshift-ca\") pod \"5c402cae-28ff-4bfc-a791-c62fc095ee20\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " Nov 04 00:50:47.983218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:47.983058 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-xks-ca\") pod \"5c402cae-28ff-4bfc-a791-c62fc095ee20\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " Nov 04 00:50:47.983218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:47.983136 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c402cae-28ff-4bfc-a791-c62fc095ee20-kube-api-access-52tj2\" (UniqueName: \"kubernetes.io/projected/5c402cae-28ff-4bfc-a791-c62fc095ee20-kube-api-access-52tj2\") pod \"5c402cae-28ff-4bfc-a791-c62fc095ee20\" (UID: \"5c402cae-28ff-4bfc-a791-c62fc095ee20\") " Nov 04 00:50:47.983349 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:47.983331 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-openshift-ca" pod "5c402cae-28ff-4bfc-a791-c62fc095ee20" (UID: "5c402cae-28ff-4bfc-a791-c62fc095ee20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:50:47.983554 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:47.983533 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:47.985104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:47.985074 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-xks-ca" pod "5c402cae-28ff-4bfc-a791-c62fc095ee20" (UID: "5c402cae-28ff-4bfc-a791-c62fc095ee20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:50:47.985245 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:47.985227 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c402cae-28ff-4bfc-a791-c62fc095ee20-kube-api-access-52tj2" pod "5c402cae-28ff-4bfc-a791-c62fc095ee20" (UID: "5c402cae-28ff-4bfc-a791-c62fc095ee20"). InnerVolumeSpecName "kube-api-access-52tj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:50:48.084973 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:48.084882 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c402cae-28ff-4bfc-a791-c62fc095ee20-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:48.084973 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:48.084920 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52tj2\" (UniqueName: \"kubernetes.io/projected/5c402cae-28ff-4bfc-a791-c62fc095ee20-kube-api-access-52tj2\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:48.811619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:48.811584 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" Nov 04 00:50:48.811871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:48.811612 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf" event={"ID":"5c402cae-28ff-4bfc-a791-c62fc095ee20","Type":"ContainerDied","Data":"585a8e230d6bfe0b215d89aa3245f17986c705547c0be2c0f88056a77ff7fe84"} Nov 04 00:50:48.811871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:48.811672 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="585a8e230d6bfe0b215d89aa3245f17986c705547c0be2c0f88056a77ff7fe84" Nov 04 00:50:54.618108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.618061 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k"] Nov 04 00:50:54.618642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.618544 6208 cpu_manager.go:414] "containerMap: removing container" podUID="5c402cae-28ff-4bfc-a791-c62fc095ee20" containerName="revoke-keys" Nov 04 00:50:54.618642 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.618563 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c402cae-28ff-4bfc-a791-c62fc095ee20" containerName="revoke-keys" Nov 04 00:50:54.618732 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.618656 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c402cae-28ff-4bfc-a791-c62fc095ee20" containerName="revoke-keys" Nov 04 00:50:54.621939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.621918 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.626940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.626916 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:50:54.628439 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.628321 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:50:54.628439 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.628356 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:50:54.628630 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.628409 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:50:54.639897 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.639868 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k"] Nov 04 00:50:54.749907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.749874 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxdt5\" (UniqueName: \"kubernetes.io/projected/c82cf639-f285-4285-a5bb-8de2e947e3bb-kube-api-access-bxdt5\") pod \"maas-api-revoke-sub-d3fba056-prm4k\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.750126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.749927 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-prm4k\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.750126 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.750006 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-prm4k\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.850802 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.850760 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-prm4k\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.851009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.850884 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxdt5\" (UniqueName: \"kubernetes.io/projected/c82cf639-f285-4285-a5bb-8de2e947e3bb-kube-api-access-bxdt5\") pod \"maas-api-revoke-sub-d3fba056-prm4k\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.851009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.850940 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-prm4k\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.851605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.851578 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-prm4k\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.853136 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.853114 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-prm4k\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.863480 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.863455 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxdt5\" (UniqueName: \"kubernetes.io/projected/c82cf639-f285-4285-a5bb-8de2e947e3bb-kube-api-access-bxdt5\") pod \"maas-api-revoke-sub-d3fba056-prm4k\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:54.936294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:54.936219 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:55.217161 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:55.213638 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k"] Nov 04 00:50:55.217623 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:50:55.217584 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82cf639_f285_4285_a5bb_8de2e947e3bb.slice/crio-cfeff9046b1ab99dca9e5b1242f9ca71d5af5deb3205461d3cd29ff2ee5db9b8 WatchSource:0}: Error finding container cfeff9046b1ab99dca9e5b1242f9ca71d5af5deb3205461d3cd29ff2ee5db9b8: Status 404 returned error can't find the container with id cfeff9046b1ab99dca9e5b1242f9ca71d5af5deb3205461d3cd29ff2ee5db9b8 Nov 04 00:50:55.851341 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:55.851231 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" event={"ID":"c82cf639-f285-4285-a5bb-8de2e947e3bb","Type":"ContainerStarted","Data":"b95b8ef465932c2f9ea7a1ee1ca3e11426963ea69fb7a2da6de7460a01a205e5"} Nov 04 00:50:55.851341 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:55.851296 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" event={"ID":"c82cf639-f285-4285-a5bb-8de2e947e3bb","Type":"ContainerStarted","Data":"cfeff9046b1ab99dca9e5b1242f9ca71d5af5deb3205461d3cd29ff2ee5db9b8"} Nov 04 00:50:55.871846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:55.871789 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" podStartSLOduration=1.871775973 podStartE2EDuration="1.871775973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:50:55.869169748 +0000 UTC m=+4874.691765911" watchObservedRunningTime="2026-11-04 00:50:55.871775973 +0000 UTC m=+4874.694372134" Nov 04 00:50:56.891953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:56.891916 6208 generic.go:356] "Generic (PLEG): container finished" podID="c82cf639-f285-4285-a5bb-8de2e947e3bb" containerID="b95b8ef465932c2f9ea7a1ee1ca3e11426963ea69fb7a2da6de7460a01a205e5" exitCode=0 Nov 04 00:50:56.902025 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:56.898072 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" event={"ID":"c82cf639-f285-4285-a5bb-8de2e947e3bb","Type":"ContainerDied","Data":"b95b8ef465932c2f9ea7a1ee1ca3e11426963ea69fb7a2da6de7460a01a205e5"} Nov 04 00:50:57.088057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:57.057111 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws"] Nov 04 00:50:57.133567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:57.116516 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rljws"] Nov 04 00:50:57.321070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:57.309306 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e42d9c-c65a-4eab-8bc7-e2b129811a38" path="/var/lib/kubelet/pods/d6e42d9c-c65a-4eab-8bc7-e2b129811a38/volumes" Nov 04 00:50:58.228016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.227985 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:58.243531 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.243502 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82cf639-f285-4285-a5bb-8de2e947e3bb-kube-api-access-bxdt5\" (UniqueName: \"kubernetes.io/projected/c82cf639-f285-4285-a5bb-8de2e947e3bb-kube-api-access-bxdt5\") pod \"c82cf639-f285-4285-a5bb-8de2e947e3bb\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " Nov 04 00:50:58.243686 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.243575 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-openshift-ca\") pod \"c82cf639-f285-4285-a5bb-8de2e947e3bb\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " Nov 04 00:50:58.244092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.244060 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-openshift-ca" pod "c82cf639-f285-4285-a5bb-8de2e947e3bb" (UID: "c82cf639-f285-4285-a5bb-8de2e947e3bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:50:58.246125 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.246091 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82cf639-f285-4285-a5bb-8de2e947e3bb-kube-api-access-bxdt5" pod "c82cf639-f285-4285-a5bb-8de2e947e3bb" (UID: "c82cf639-f285-4285-a5bb-8de2e947e3bb"). InnerVolumeSpecName "kube-api-access-bxdt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:50:58.344044 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.344009 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-xks-ca\") pod \"c82cf639-f285-4285-a5bb-8de2e947e3bb\" (UID: \"c82cf639-f285-4285-a5bb-8de2e947e3bb\") " Nov 04 00:50:58.344536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.344514 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxdt5\" (UniqueName: \"kubernetes.io/projected/c82cf639-f285-4285-a5bb-8de2e947e3bb-kube-api-access-bxdt5\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:58.344655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.344541 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:58.346216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.346190 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-xks-ca" pod "c82cf639-f285-4285-a5bb-8de2e947e3bb" (UID: "c82cf639-f285-4285-a5bb-8de2e947e3bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:50:58.445432 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.445405 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cf639-f285-4285-a5bb-8de2e947e3bb-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:50:58.823411 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.823374 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:50:58.923905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.923873 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" Nov 04 00:50:58.924109 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.923910 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k" event={"ID":"c82cf639-f285-4285-a5bb-8de2e947e3bb","Type":"ContainerDied","Data":"cfeff9046b1ab99dca9e5b1242f9ca71d5af5deb3205461d3cd29ff2ee5db9b8"} Nov 04 00:50:58.924109 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:50:58.923959 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfeff9046b1ab99dca9e5b1242f9ca71d5af5deb3205461d3cd29ff2ee5db9b8" Nov 04 00:51:05.706752 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:51:05.706709 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:51:15.451062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.450999 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n"] Nov 04 00:51:15.451618 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.451575 6208 cpu_manager.go:414] "containerMap: removing container" podUID="c82cf639-f285-4285-a5bb-8de2e947e3bb" containerName="revoke-keys" Nov 04 00:51:15.451618 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.451597 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82cf639-f285-4285-a5bb-8de2e947e3bb" containerName="revoke-keys" Nov 04 00:51:15.451773 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.451757 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82cf639-f285-4285-a5bb-8de2e947e3bb" containerName="revoke-keys" Nov 04 00:51:15.455125 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.455103 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.462147 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.462120 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:51:15.462430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.462410 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:51:15.464116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.462607 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:51:15.464116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.462678 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:51:15.465528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.465501 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n"] Nov 04 00:51:15.503365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.503333 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mc4s\" (UniqueName: \"kubernetes.io/projected/54de174b-a2c7-4668-8ba9-355a1783b1ef-kube-api-access-6mc4s\") pod \"maas-api-revoke-sub-ae594979-k758n\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.503549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.503380 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-k758n\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.503549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.503501 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-k758n\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.605054 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.605003 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mc4s\" (UniqueName: \"kubernetes.io/projected/54de174b-a2c7-4668-8ba9-355a1783b1ef-kube-api-access-6mc4s\") pod \"maas-api-revoke-sub-ae594979-k758n\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.605210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.605068 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-k758n\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.605210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.605102 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-k758n\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.605735 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.605713 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-k758n\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.607487 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.607462 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-k758n\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.617527 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.617505 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mc4s\" (UniqueName: \"kubernetes.io/projected/54de174b-a2c7-4668-8ba9-355a1783b1ef-kube-api-access-6mc4s\") pod \"maas-api-revoke-sub-ae594979-k758n\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:15.770706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:15.770674 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:16.046694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:16.046658 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n"] Nov 04 00:51:16.048145 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:51:16.048107 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54de174b_a2c7_4668_8ba9_355a1783b1ef.slice/crio-18e25a8d9ac9abac7d1a7a9c6623a311c35bf52b0d84f21fe88ea227af66d55c WatchSource:0}: Error finding container 18e25a8d9ac9abac7d1a7a9c6623a311c35bf52b0d84f21fe88ea227af66d55c: Status 404 returned error can't find the container with id 18e25a8d9ac9abac7d1a7a9c6623a311c35bf52b0d84f21fe88ea227af66d55c Nov 04 00:51:17.059379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:17.059348 6208 generic.go:356] "Generic (PLEG): container finished" podID="54de174b-a2c7-4668-8ba9-355a1783b1ef" containerID="f47d5ba904a5e66260cc75181902c0eb2e1a86af97298b0299a52950264c6500" exitCode=0 Nov 04 00:51:17.059787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:17.059445 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" event={"ID":"54de174b-a2c7-4668-8ba9-355a1783b1ef","Type":"ContainerDied","Data":"f47d5ba904a5e66260cc75181902c0eb2e1a86af97298b0299a52950264c6500"} Nov 04 00:51:17.059787 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:17.059498 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" event={"ID":"54de174b-a2c7-4668-8ba9-355a1783b1ef","Type":"ContainerStarted","Data":"18e25a8d9ac9abac7d1a7a9c6623a311c35bf52b0d84f21fe88ea227af66d55c"} Nov 04 00:51:18.201756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.201728 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:18.231234 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.231203 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54de174b-a2c7-4668-8ba9-355a1783b1ef-kube-api-access-6mc4s\" (UniqueName: \"kubernetes.io/projected/54de174b-a2c7-4668-8ba9-355a1783b1ef-kube-api-access-6mc4s\") pod \"54de174b-a2c7-4668-8ba9-355a1783b1ef\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " Nov 04 00:51:18.231389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.231296 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-xks-ca\") pod \"54de174b-a2c7-4668-8ba9-355a1783b1ef\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " Nov 04 00:51:18.231389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.231335 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-openshift-ca\") pod \"54de174b-a2c7-4668-8ba9-355a1783b1ef\" (UID: \"54de174b-a2c7-4668-8ba9-355a1783b1ef\") " Nov 04 00:51:18.231762 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.231721 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-openshift-ca" pod "54de174b-a2c7-4668-8ba9-355a1783b1ef" (UID: "54de174b-a2c7-4668-8ba9-355a1783b1ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:51:18.233518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.233484 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-xks-ca" pod "54de174b-a2c7-4668-8ba9-355a1783b1ef" (UID: "54de174b-a2c7-4668-8ba9-355a1783b1ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:51:18.233636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.233528 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54de174b-a2c7-4668-8ba9-355a1783b1ef-kube-api-access-6mc4s" pod "54de174b-a2c7-4668-8ba9-355a1783b1ef" (UID: "54de174b-a2c7-4668-8ba9-355a1783b1ef"). InnerVolumeSpecName "kube-api-access-6mc4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:51:18.332398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.332319 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:51:18.332398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.332348 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54de174b-a2c7-4668-8ba9-355a1783b1ef-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:51:18.332398 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:18.332360 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mc4s\" (UniqueName: \"kubernetes.io/projected/54de174b-a2c7-4668-8ba9-355a1783b1ef-kube-api-access-6mc4s\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:51:19.070352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:19.070313 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" event={"ID":"54de174b-a2c7-4668-8ba9-355a1783b1ef","Type":"ContainerDied","Data":"18e25a8d9ac9abac7d1a7a9c6623a311c35bf52b0d84f21fe88ea227af66d55c"} Nov 04 00:51:19.070352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:19.070332 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n" Nov 04 00:51:19.070583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:19.070368 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e25a8d9ac9abac7d1a7a9c6623a311c35bf52b0d84f21fe88ea227af66d55c" Nov 04 00:51:20.320737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:20.320697 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:51:22.026689 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:22.026654 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k"] Nov 04 00:51:22.028650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:22.028625 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsq8k"] Nov 04 00:51:23.293419 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:23.293373 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd1d263-012f-4dbf-babc-b0af5b450ea3" path="/var/lib/kubelet/pods/4bd1d263-012f-4dbf-babc-b0af5b450ea3/volumes" Nov 04 00:51:24.770940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:24.770900 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:51:30.659242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:30.659192 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:51:30.660018 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:30.659974 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:51:31.729795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.729754 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld"] Nov 04 00:51:31.730318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.730170 6208 cpu_manager.go:414] "containerMap: removing container" podUID="54de174b-a2c7-4668-8ba9-355a1783b1ef" containerName="revoke-keys" Nov 04 00:51:31.730318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.730183 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54de174b-a2c7-4668-8ba9-355a1783b1ef" containerName="revoke-keys" Nov 04 00:51:31.730318 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.730257 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="54de174b-a2c7-4668-8ba9-355a1783b1ef" containerName="revoke-keys" Nov 04 00:51:31.733323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.733304 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:31.741343 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.741313 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:51:31.741477 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.741356 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:51:31.741477 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.741396 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:51:31.741477 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.741470 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:51:31.744908 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.744882 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld"] Nov 04 00:51:31.778406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.778373 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e991557-468c-4e3c-8068-29286832e39d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-69gld\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:31.778583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.778415 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2c5\" (UniqueName: \"kubernetes.io/projected/8e991557-468c-4e3c-8068-29286832e39d-kube-api-access-zh2c5\") pod \"maas-api-revoke-sub-a4aef7d7-69gld\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:31.778583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.778565 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e991557-468c-4e3c-8068-29286832e39d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-69gld\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:31.879627 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.879591 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e991557-468c-4e3c-8068-29286832e39d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-69gld\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:31.879808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.879641 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e991557-468c-4e3c-8068-29286832e39d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-69gld\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:31.879808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.879671 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2c5\" (UniqueName: \"kubernetes.io/projected/8e991557-468c-4e3c-8068-29286832e39d-kube-api-access-zh2c5\") pod \"maas-api-revoke-sub-a4aef7d7-69gld\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:31.880337 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.880312 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e991557-468c-4e3c-8068-29286832e39d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-69gld\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:31.882064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.882020 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e991557-468c-4e3c-8068-29286832e39d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-69gld\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:31.891850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:31.891828 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2c5\" (UniqueName: \"kubernetes.io/projected/8e991557-468c-4e3c-8068-29286832e39d-kube-api-access-zh2c5\") pod \"maas-api-revoke-sub-a4aef7d7-69gld\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:32.050261 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:32.050165 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:32.326993 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:32.326960 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld"] Nov 04 00:51:32.328516 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:51:32.328478 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e991557_468c_4e3c_8068_29286832e39d.slice/crio-7aba7fa691502d0fbd1e70921ec799d76da6446c0705c07fef9e21ef8614ca5a WatchSource:0}: Error finding container 7aba7fa691502d0fbd1e70921ec799d76da6446c0705c07fef9e21ef8614ca5a: Status 404 returned error can't find the container with id 7aba7fa691502d0fbd1e70921ec799d76da6446c0705c07fef9e21ef8614ca5a Nov 04 00:51:33.154822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:33.154782 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" event={"ID":"8e991557-468c-4e3c-8068-29286832e39d","Type":"ContainerStarted","Data":"9a848c9074104ba03b38ff85a621b93e91758bb08fba5ff2e0b4bbadb7b51eb2"} Nov 04 00:51:33.155307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:33.154838 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" event={"ID":"8e991557-468c-4e3c-8068-29286832e39d","Type":"ContainerStarted","Data":"7aba7fa691502d0fbd1e70921ec799d76da6446c0705c07fef9e21ef8614ca5a"} Nov 04 00:51:33.175810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:33.175760 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" podStartSLOduration=2.1757442559999998 podStartE2EDuration="2.175744256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:51:33.173496165 +0000 UTC m=+4911.996092329" watchObservedRunningTime="2026-11-04 00:51:33.175744256 +0000 UTC m=+4911.998340426" Nov 04 00:51:34.163982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:34.163940 6208 generic.go:356] "Generic (PLEG): container finished" podID="8e991557-468c-4e3c-8068-29286832e39d" containerID="9a848c9074104ba03b38ff85a621b93e91758bb08fba5ff2e0b4bbadb7b51eb2" exitCode=0 Nov 04 00:51:34.164495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:34.164044 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" event={"ID":"8e991557-468c-4e3c-8068-29286832e39d","Type":"ContainerDied","Data":"9a848c9074104ba03b38ff85a621b93e91758bb08fba5ff2e0b4bbadb7b51eb2"} Nov 04 00:51:35.300632 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.300605 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:35.414858 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.414820 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e991557-468c-4e3c-8068-29286832e39d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e991557-468c-4e3c-8068-29286832e39d-maas-api-openshift-ca\") pod \"8e991557-468c-4e3c-8068-29286832e39d\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " Nov 04 00:51:35.415053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.414909 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e991557-468c-4e3c-8068-29286832e39d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e991557-468c-4e3c-8068-29286832e39d-maas-api-xks-ca\") pod \"8e991557-468c-4e3c-8068-29286832e39d\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " Nov 04 00:51:35.415053 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.414976 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e991557-468c-4e3c-8068-29286832e39d-kube-api-access-zh2c5\" (UniqueName: \"kubernetes.io/projected/8e991557-468c-4e3c-8068-29286832e39d-kube-api-access-zh2c5\") pod \"8e991557-468c-4e3c-8068-29286832e39d\" (UID: \"8e991557-468c-4e3c-8068-29286832e39d\") " Nov 04 00:51:35.415275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.415244 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e991557-468c-4e3c-8068-29286832e39d-maas-api-openshift-ca" pod "8e991557-468c-4e3c-8068-29286832e39d" (UID: "8e991557-468c-4e3c-8068-29286832e39d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:51:35.416948 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.416920 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e991557-468c-4e3c-8068-29286832e39d-kube-api-access-zh2c5" pod "8e991557-468c-4e3c-8068-29286832e39d" (UID: "8e991557-468c-4e3c-8068-29286832e39d"). InnerVolumeSpecName "kube-api-access-zh2c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:51:35.417050 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.416952 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e991557-468c-4e3c-8068-29286832e39d-maas-api-xks-ca" pod "8e991557-468c-4e3c-8068-29286832e39d" (UID: "8e991557-468c-4e3c-8068-29286832e39d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:51:35.516285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.516251 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh2c5\" (UniqueName: \"kubernetes.io/projected/8e991557-468c-4e3c-8068-29286832e39d-kube-api-access-zh2c5\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:51:35.516285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.516281 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e991557-468c-4e3c-8068-29286832e39d-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:51:35.516460 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.516292 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e991557-468c-4e3c-8068-29286832e39d-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:51:35.703122 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:51:35.703029 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:51:35.940046 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:35.940013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:51:36.038304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:36.038274 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:51:36.175570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:36.175536 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" event={"ID":"8e991557-468c-4e3c-8068-29286832e39d","Type":"ContainerDied","Data":"7aba7fa691502d0fbd1e70921ec799d76da6446c0705c07fef9e21ef8614ca5a"} Nov 04 00:51:36.175746 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:36.175585 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aba7fa691502d0fbd1e70921ec799d76da6446c0705c07fef9e21ef8614ca5a" Nov 04 00:51:36.175746 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:36.175550 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld" Nov 04 00:51:39.027601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:39.027406 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh"] Nov 04 00:51:39.029907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:39.029879 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd409a3-rs9rh"] Nov 04 00:51:39.293607 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:39.293531 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4d1e80-2803-4006-834e-96f745f91016" path="/var/lib/kubelet/pods/0b4d1e80-2803-4006-834e-96f745f91016/volumes" Nov 04 00:51:44.665622 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:44.665590 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:51:47.545697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.545187 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:51:47.545697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.545264 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:51:47.568175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.568138 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk"] Nov 04 00:51:47.569001 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.568972 6208 cpu_manager.go:414] "containerMap: removing container" podUID="8e991557-468c-4e3c-8068-29286832e39d" containerName="revoke-keys" Nov 04 00:51:47.569197 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.569179 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e991557-468c-4e3c-8068-29286832e39d" containerName="revoke-keys" Nov 04 00:51:47.569401 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.569389 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e991557-468c-4e3c-8068-29286832e39d" containerName="revoke-keys" Nov 04 00:51:47.590062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.588729 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk"] Nov 04 00:51:47.590062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.588849 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:47.594063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.594018 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:51:47.594214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.594168 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:51:47.594287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.594243 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:51:47.598063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.597195 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:51:47.748917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.748865 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wbssk\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:47.749263 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.749223 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wbssk\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:47.749533 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.749507 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnmx9\" (UniqueName: \"kubernetes.io/projected/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-kube-api-access-gnmx9\") pod \"maas-api-revoke-sub-79805ca9-wbssk\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:47.850450 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.850351 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wbssk\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:47.850450 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.850393 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wbssk\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:47.850450 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.850413 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnmx9\" (UniqueName: \"kubernetes.io/projected/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-kube-api-access-gnmx9\") pod \"maas-api-revoke-sub-79805ca9-wbssk\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:47.851607 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.851576 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wbssk\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:47.854570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.854543 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wbssk\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:47.863156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:47.863128 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnmx9\" (UniqueName: \"kubernetes.io/projected/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-kube-api-access-gnmx9\") pod \"maas-api-revoke-sub-79805ca9-wbssk\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:48.022851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:48.022809 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:48.297124 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:48.297093 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk"] Nov 04 00:51:48.299391 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:51:48.299363 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b53f9e4_b9f1_40ce_8731_9ddfd9d241a3.slice/crio-8440c3eaeac18480988acdda914cf762299d724101f76dfcff559b923332f309 WatchSource:0}: Error finding container 8440c3eaeac18480988acdda914cf762299d724101f76dfcff559b923332f309: Status 404 returned error can't find the container with id 8440c3eaeac18480988acdda914cf762299d724101f76dfcff559b923332f309 Nov 04 00:51:49.258573 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:49.258537 6208 generic.go:356] "Generic (PLEG): container finished" podID="4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3" containerID="ddb7a757bb8db4f89845b79139af2552b6fbcfbf20e7159114aa3efe9c02677b" exitCode=0 Nov 04 00:51:49.258967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:49.258628 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" event={"ID":"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3","Type":"ContainerDied","Data":"ddb7a757bb8db4f89845b79139af2552b6fbcfbf20e7159114aa3efe9c02677b"} Nov 04 00:51:49.258967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:49.258689 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" event={"ID":"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3","Type":"ContainerStarted","Data":"8440c3eaeac18480988acdda914cf762299d724101f76dfcff559b923332f309"} Nov 04 00:51:50.395082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.395058 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:50.472456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.472423 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-openshift-ca\") pod \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " Nov 04 00:51:50.472598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.472498 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-kube-api-access-gnmx9\" (UniqueName: \"kubernetes.io/projected/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-kube-api-access-gnmx9\") pod \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " Nov 04 00:51:50.472598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.472558 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-xks-ca\") pod \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\" (UID: \"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3\") " Nov 04 00:51:50.472856 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.472831 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-openshift-ca" pod "4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3" (UID: "4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:51:50.474582 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.474558 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-xks-ca" pod "4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3" (UID: "4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:51:50.474716 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.474694 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-kube-api-access-gnmx9" pod "4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3" (UID: "4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3"). InnerVolumeSpecName "kube-api-access-gnmx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:51:50.574156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.574071 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:51:50.574156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.574107 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:51:50.574156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:50.574121 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnmx9\" (UniqueName: \"kubernetes.io/projected/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3-kube-api-access-gnmx9\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:51:51.270163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:51.270131 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" Nov 04 00:51:51.270163 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:51.270147 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk" event={"ID":"4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3","Type":"ContainerDied","Data":"8440c3eaeac18480988acdda914cf762299d724101f76dfcff559b923332f309"} Nov 04 00:51:51.270378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:51.270193 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8440c3eaeac18480988acdda914cf762299d724101f76dfcff559b923332f309" Nov 04 00:51:59.003523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:51:59.003472 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:52:02.279058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.275834 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd"] Nov 04 00:52:02.279058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.276478 6208 cpu_manager.go:414] "containerMap: removing container" podUID="4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3" containerName="revoke-keys" Nov 04 00:52:02.279058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.276501 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3" containerName="revoke-keys" Nov 04 00:52:02.279058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.276627 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3" containerName="revoke-keys" Nov 04 00:52:02.282069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.280181 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.291059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.288477 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:52:02.291059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.289216 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:52:02.291059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.289297 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:52:02.291059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.289366 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:52:02.291059 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.290051 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd"] Nov 04 00:52:02.424922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.424612 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tqhgd\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.424922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.424691 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tqhgd\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.424922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.424723 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpjhl\" (UniqueName: \"kubernetes.io/projected/9ceb16cd-fbaa-4904-a7af-593e33494564-kube-api-access-bpjhl\") pod \"maas-api-revoke-sub-0b7e8edb-tqhgd\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.526529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.526471 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpjhl\" (UniqueName: \"kubernetes.io/projected/9ceb16cd-fbaa-4904-a7af-593e33494564-kube-api-access-bpjhl\") pod \"maas-api-revoke-sub-0b7e8edb-tqhgd\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.526805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.526708 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tqhgd\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.526805 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.526767 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tqhgd\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.528098 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.528066 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tqhgd\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.532516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.532449 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tqhgd\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.540051 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.540011 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpjhl\" (UniqueName: \"kubernetes.io/projected/9ceb16cd-fbaa-4904-a7af-593e33494564-kube-api-access-bpjhl\") pod \"maas-api-revoke-sub-0b7e8edb-tqhgd\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:02.650160 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:02.650114 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:03.125665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:03.124204 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd"] Nov 04 00:52:03.126055 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:52:03.125984 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ceb16cd_fbaa_4904_a7af_593e33494564.slice/crio-2a3e7cb734133b777f9b0aea802399a1ad2436f948eb16b56fb9d0b98a4a40ef WatchSource:0}: Error finding container 2a3e7cb734133b777f9b0aea802399a1ad2436f948eb16b56fb9d0b98a4a40ef: Status 404 returned error can't find the container with id 2a3e7cb734133b777f9b0aea802399a1ad2436f948eb16b56fb9d0b98a4a40ef Nov 04 00:52:03.236238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:03.236192 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-5hv2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 00:52:03.236416 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:03.236243 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-5hv2p" podUID="de2726d5-efb3-4832-977b-e2315e48a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 00:52:03.353351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:03.353245 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" event={"ID":"9ceb16cd-fbaa-4904-a7af-593e33494564","Type":"ContainerStarted","Data":"2a3e7cb734133b777f9b0aea802399a1ad2436f948eb16b56fb9d0b98a4a40ef"} Nov 04 00:52:04.361541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:04.361446 6208 generic.go:356] "Generic (PLEG): container finished" podID="9ceb16cd-fbaa-4904-a7af-593e33494564" containerID="00b29a322c094e8232887e5e4ba5b22e702f2196107161bee1c11142cd7ad6b3" exitCode=0 Nov 04 00:52:04.361541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:04.361530 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" event={"ID":"9ceb16cd-fbaa-4904-a7af-593e33494564","Type":"ContainerDied","Data":"00b29a322c094e8232887e5e4ba5b22e702f2196107161bee1c11142cd7ad6b3"} Nov 04 00:52:05.505440 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.505414 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:05.565767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.565737 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-xks-ca\") pod \"9ceb16cd-fbaa-4904-a7af-593e33494564\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " Nov 04 00:52:05.565911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.565788 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-openshift-ca\") pod \"9ceb16cd-fbaa-4904-a7af-593e33494564\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " Nov 04 00:52:05.565911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.565827 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ceb16cd-fbaa-4904-a7af-593e33494564-kube-api-access-bpjhl\" (UniqueName: \"kubernetes.io/projected/9ceb16cd-fbaa-4904-a7af-593e33494564-kube-api-access-bpjhl\") pod \"9ceb16cd-fbaa-4904-a7af-593e33494564\" (UID: \"9ceb16cd-fbaa-4904-a7af-593e33494564\") " Nov 04 00:52:05.566239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.566213 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-openshift-ca" pod "9ceb16cd-fbaa-4904-a7af-593e33494564" (UID: "9ceb16cd-fbaa-4904-a7af-593e33494564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:52:05.567748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.567719 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-xks-ca" pod "9ceb16cd-fbaa-4904-a7af-593e33494564" (UID: "9ceb16cd-fbaa-4904-a7af-593e33494564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:52:05.567839 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.567824 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ceb16cd-fbaa-4904-a7af-593e33494564-kube-api-access-bpjhl" pod "9ceb16cd-fbaa-4904-a7af-593e33494564" (UID: "9ceb16cd-fbaa-4904-a7af-593e33494564"). InnerVolumeSpecName "kube-api-access-bpjhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:52:05.667543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.667461 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:52:05.667543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.667492 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpjhl\" (UniqueName: \"kubernetes.io/projected/9ceb16cd-fbaa-4904-a7af-593e33494564-kube-api-access-bpjhl\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:52:05.667543 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:05.667502 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ceb16cd-fbaa-4904-a7af-593e33494564-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:52:05.705129 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:52:05.705093 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:52:06.374659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:06.374628 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" Nov 04 00:52:06.374866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:06.374664 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd" event={"ID":"9ceb16cd-fbaa-4904-a7af-593e33494564","Type":"ContainerDied","Data":"2a3e7cb734133b777f9b0aea802399a1ad2436f948eb16b56fb9d0b98a4a40ef"} Nov 04 00:52:06.374866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:06.374713 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a3e7cb734133b777f9b0aea802399a1ad2436f948eb16b56fb9d0b98a4a40ef" Nov 04 00:52:12.956850 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.956820 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz"] Nov 04 00:52:12.957298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.957280 6208 cpu_manager.go:414] "containerMap: removing container" podUID="9ceb16cd-fbaa-4904-a7af-593e33494564" containerName="revoke-keys" Nov 04 00:52:12.957378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.957300 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ceb16cd-fbaa-4904-a7af-593e33494564" containerName="revoke-keys" Nov 04 00:52:12.957378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.957361 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ceb16cd-fbaa-4904-a7af-593e33494564" containerName="revoke-keys" Nov 04 00:52:12.960637 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.960612 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:12.969320 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.969296 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:52:12.970817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.970792 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:52:12.970946 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.970873 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:52:12.970946 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.970920 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:52:12.972309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:12.972287 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz"] Nov 04 00:52:13.039345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.039311 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h572w\" (UniqueName: \"kubernetes.io/projected/ae3cbdd9-570d-498b-a400-380f89fbaa0e-kube-api-access-h572w\") pod \"maas-api-revoke-sub-3d4156f1-dl7jz\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.039345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.039351 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dl7jz\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.039560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.039376 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dl7jz\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.140605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.140573 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h572w\" (UniqueName: \"kubernetes.io/projected/ae3cbdd9-570d-498b-a400-380f89fbaa0e-kube-api-access-h572w\") pod \"maas-api-revoke-sub-3d4156f1-dl7jz\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.140775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.140609 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dl7jz\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.140775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.140637 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dl7jz\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.143876 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.141863 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dl7jz\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.144639 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.143990 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dl7jz\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.153515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.153492 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h572w\" (UniqueName: \"kubernetes.io/projected/ae3cbdd9-570d-498b-a400-380f89fbaa0e-kube-api-access-h572w\") pod \"maas-api-revoke-sub-3d4156f1-dl7jz\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.276175 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.276109 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:13.555269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:13.555239 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz"] Nov 04 00:52:13.557172 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:52:13.557142 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3cbdd9_570d_498b_a400_380f89fbaa0e.slice/crio-c25f8c3bde7510004dff414eea582a3c2a0829bebf4e8d3bd45aa87d082efa01 WatchSource:0}: Error finding container c25f8c3bde7510004dff414eea582a3c2a0829bebf4e8d3bd45aa87d082efa01: Status 404 returned error can't find the container with id c25f8c3bde7510004dff414eea582a3c2a0829bebf4e8d3bd45aa87d082efa01 Nov 04 00:52:14.424567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:14.424531 6208 generic.go:356] "Generic (PLEG): container finished" podID="ae3cbdd9-570d-498b-a400-380f89fbaa0e" containerID="fd2097b486f182dd16a6128afe69811a56d6a6b0bdd7b3de10c2978e729edf68" exitCode=0 Nov 04 00:52:14.424965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:14.424584 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" event={"ID":"ae3cbdd9-570d-498b-a400-380f89fbaa0e","Type":"ContainerDied","Data":"fd2097b486f182dd16a6128afe69811a56d6a6b0bdd7b3de10c2978e729edf68"} Nov 04 00:52:14.424965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:14.424622 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" event={"ID":"ae3cbdd9-570d-498b-a400-380f89fbaa0e","Type":"ContainerStarted","Data":"c25f8c3bde7510004dff414eea582a3c2a0829bebf4e8d3bd45aa87d082efa01"} Nov 04 00:52:15.563683 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.563655 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:15.664207 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.664166 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-openshift-ca\") pod \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " Nov 04 00:52:15.664424 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.664228 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3cbdd9-570d-498b-a400-380f89fbaa0e-kube-api-access-h572w\" (UniqueName: \"kubernetes.io/projected/ae3cbdd9-570d-498b-a400-380f89fbaa0e-kube-api-access-h572w\") pod \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " Nov 04 00:52:15.664424 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.664252 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-xks-ca\") pod \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\" (UID: \"ae3cbdd9-570d-498b-a400-380f89fbaa0e\") " Nov 04 00:52:15.664636 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.664608 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-openshift-ca" pod "ae3cbdd9-570d-498b-a400-380f89fbaa0e" (UID: "ae3cbdd9-570d-498b-a400-380f89fbaa0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:52:15.666461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.666429 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-xks-ca" pod "ae3cbdd9-570d-498b-a400-380f89fbaa0e" (UID: "ae3cbdd9-570d-498b-a400-380f89fbaa0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:52:15.666461 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.666437 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3cbdd9-570d-498b-a400-380f89fbaa0e-kube-api-access-h572w" pod "ae3cbdd9-570d-498b-a400-380f89fbaa0e" (UID: "ae3cbdd9-570d-498b-a400-380f89fbaa0e"). InnerVolumeSpecName "kube-api-access-h572w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:52:15.765948 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.765922 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:52:15.765948 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.765947 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h572w\" (UniqueName: \"kubernetes.io/projected/ae3cbdd9-570d-498b-a400-380f89fbaa0e-kube-api-access-h572w\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:52:15.766179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:15.765958 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3cbdd9-570d-498b-a400-380f89fbaa0e-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:52:16.438103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:16.438068 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" Nov 04 00:52:16.438103 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:16.438088 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz" event={"ID":"ae3cbdd9-570d-498b-a400-380f89fbaa0e","Type":"ContainerDied","Data":"c25f8c3bde7510004dff414eea582a3c2a0829bebf4e8d3bd45aa87d082efa01"} Nov 04 00:52:16.438331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:16.438146 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c25f8c3bde7510004dff414eea582a3c2a0829bebf4e8d3bd45aa87d082efa01" Nov 04 00:52:20.380699 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:20.380617 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:52:24.836592 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:24.836547 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:52:35.707244 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:52:35.707196 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:52:38.002496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.002454 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz"] Nov 04 00:52:38.006617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.006584 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dl7jz"] Nov 04 00:52:38.026236 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.026194 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45"] Nov 04 00:52:38.026619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.026604 6208 cpu_manager.go:414] "containerMap: removing container" podUID="ae3cbdd9-570d-498b-a400-380f89fbaa0e" containerName="revoke-keys" Nov 04 00:52:38.026619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.026619 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3cbdd9-570d-498b-a400-380f89fbaa0e" containerName="revoke-keys" Nov 04 00:52:38.026775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.026702 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3cbdd9-570d-498b-a400-380f89fbaa0e" containerName="revoke-keys" Nov 04 00:52:38.030006 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.029985 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.037634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.037606 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:52:38.037634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.037625 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:52:38.037781 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.037710 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:52:38.038216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.037713 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:52:38.041962 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.041937 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45"] Nov 04 00:52:38.140736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.140700 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4p45\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.140914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.140770 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4p45\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.140914 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.140852 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgnhr\" (UniqueName: \"kubernetes.io/projected/6b8ef494-5763-4ec9-b1c7-2918f560b943-kube-api-access-wgnhr\") pod \"maas-api-revoke-sub-3d4156f1-m4p45\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.241700 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.241662 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4p45\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.241895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.241727 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4p45\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.241895 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.241778 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgnhr\" (UniqueName: \"kubernetes.io/projected/6b8ef494-5763-4ec9-b1c7-2918f560b943-kube-api-access-wgnhr\") pod \"maas-api-revoke-sub-3d4156f1-m4p45\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.242394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.242365 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4p45\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.244212 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.244181 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4p45\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.253925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.253869 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgnhr\" (UniqueName: \"kubernetes.io/projected/6b8ef494-5763-4ec9-b1c7-2918f560b943-kube-api-access-wgnhr\") pod \"maas-api-revoke-sub-3d4156f1-m4p45\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.347977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.347948 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:38.629646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:38.629616 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45"] Nov 04 00:52:38.631893 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:52:38.631857 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8ef494_5763_4ec9_b1c7_2918f560b943.slice/crio-3f66bf177fef2290d863cbd96b4b0ba173af1b0dbcffb842277804c18837ea0a WatchSource:0}: Error finding container 3f66bf177fef2290d863cbd96b4b0ba173af1b0dbcffb842277804c18837ea0a: Status 404 returned error can't find the container with id 3f66bf177fef2290d863cbd96b4b0ba173af1b0dbcffb842277804c18837ea0a Nov 04 00:52:39.293134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:39.293060 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3cbdd9-570d-498b-a400-380f89fbaa0e" path="/var/lib/kubelet/pods/ae3cbdd9-570d-498b-a400-380f89fbaa0e/volumes" Nov 04 00:52:39.624135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:39.624049 6208 generic.go:356] "Generic (PLEG): container finished" podID="6b8ef494-5763-4ec9-b1c7-2918f560b943" containerID="35354d453ad07d28cbe9fa8c38af34784ca8a427f15c75ae375886ae15168154" exitCode=0 Nov 04 00:52:39.624295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:39.624136 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" event={"ID":"6b8ef494-5763-4ec9-b1c7-2918f560b943","Type":"ContainerDied","Data":"35354d453ad07d28cbe9fa8c38af34784ca8a427f15c75ae375886ae15168154"} Nov 04 00:52:39.624295 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:39.624193 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" event={"ID":"6b8ef494-5763-4ec9-b1c7-2918f560b943","Type":"ContainerStarted","Data":"3f66bf177fef2290d863cbd96b4b0ba173af1b0dbcffb842277804c18837ea0a"} Nov 04 00:52:40.763309 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.763285 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:40.863170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.863136 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-openshift-ca\") pod \"6b8ef494-5763-4ec9-b1c7-2918f560b943\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " Nov 04 00:52:40.863352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.863187 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8ef494-5763-4ec9-b1c7-2918f560b943-kube-api-access-wgnhr\" (UniqueName: \"kubernetes.io/projected/6b8ef494-5763-4ec9-b1c7-2918f560b943-kube-api-access-wgnhr\") pod \"6b8ef494-5763-4ec9-b1c7-2918f560b943\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " Nov 04 00:52:40.863352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.863260 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-xks-ca\") pod \"6b8ef494-5763-4ec9-b1c7-2918f560b943\" (UID: \"6b8ef494-5763-4ec9-b1c7-2918f560b943\") " Nov 04 00:52:40.863523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.863499 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-openshift-ca" pod "6b8ef494-5763-4ec9-b1c7-2918f560b943" (UID: "6b8ef494-5763-4ec9-b1c7-2918f560b943"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:52:40.865304 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.865269 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-xks-ca" pod "6b8ef494-5763-4ec9-b1c7-2918f560b943" (UID: "6b8ef494-5763-4ec9-b1c7-2918f560b943"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:52:40.865422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.865400 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8ef494-5763-4ec9-b1c7-2918f560b943-kube-api-access-wgnhr" pod "6b8ef494-5763-4ec9-b1c7-2918f560b943" (UID: "6b8ef494-5763-4ec9-b1c7-2918f560b943"). InnerVolumeSpecName "kube-api-access-wgnhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:52:40.963978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.963946 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgnhr\" (UniqueName: \"kubernetes.io/projected/6b8ef494-5763-4ec9-b1c7-2918f560b943-kube-api-access-wgnhr\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:52:40.963978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.963979 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:52:40.964191 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:40.963990 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b8ef494-5763-4ec9-b1c7-2918f560b943-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:52:41.637511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:41.637425 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" event={"ID":"6b8ef494-5763-4ec9-b1c7-2918f560b943","Type":"ContainerDied","Data":"3f66bf177fef2290d863cbd96b4b0ba173af1b0dbcffb842277804c18837ea0a"} Nov 04 00:52:41.637511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:41.637456 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45" Nov 04 00:52:41.637511 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:41.637476 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f66bf177fef2290d863cbd96b4b0ba173af1b0dbcffb842277804c18837ea0a" Nov 04 00:52:47.243165 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:47.243129 6208 scope.go:122] "RemoveContainer" containerID="5b395421803e351108693cde47ee533e1b8bd0d7ca12de6e203224ba21c21da9" Nov 04 00:52:47.252557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:47.252530 6208 scope.go:122] "RemoveContainer" containerID="11e80bb659221d68a1d09374a9d5b23e140647b0f10f26f36bd7c28a16910329" Nov 04 00:52:47.260574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:47.260554 6208 scope.go:122] "RemoveContainer" containerID="f47a9ff62ff0e13dfb8600438215fdefdea3d7f4a6ee5a4279c7dde0565a450b" Nov 04 00:52:59.025386 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:52:59.025349 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:53:05.704166 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:53:05.704129 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:53:10.650501 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.650457 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks"] Nov 04 00:53:10.651064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.650878 6208 cpu_manager.go:414] "containerMap: removing container" podUID="6b8ef494-5763-4ec9-b1c7-2918f560b943" containerName="revoke-keys" Nov 04 00:53:10.651064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.650896 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8ef494-5763-4ec9-b1c7-2918f560b943" containerName="revoke-keys" Nov 04 00:53:10.651064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.651000 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8ef494-5763-4ec9-b1c7-2918f560b943" containerName="revoke-keys" Nov 04 00:53:10.654260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.654238 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.660625 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.660602 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:53:10.662892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.662872 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:53:10.663016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.663003 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:53:10.663115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.663086 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:53:10.666087 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.666060 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks"] Nov 04 00:53:10.752606 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.752570 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gccks\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.752793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.752614 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gccks\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.752793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.752772 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6tzv\" (UniqueName: \"kubernetes.io/projected/d5a9a396-9047-4f41-8d49-f739f0886037-kube-api-access-k6tzv\") pod \"maas-api-revoke-sub-e2fc9c5b-gccks\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.853561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.853523 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6tzv\" (UniqueName: \"kubernetes.io/projected/d5a9a396-9047-4f41-8d49-f739f0886037-kube-api-access-k6tzv\") pod \"maas-api-revoke-sub-e2fc9c5b-gccks\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.853748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.853587 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gccks\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.853748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.853624 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gccks\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.854325 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.854295 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gccks\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.856068 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.856021 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-gccks\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.866080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.866057 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6tzv\" (UniqueName: \"kubernetes.io/projected/d5a9a396-9047-4f41-8d49-f739f0886037-kube-api-access-k6tzv\") pod \"maas-api-revoke-sub-e2fc9c5b-gccks\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:10.969490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:10.969456 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:11.244883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:11.244857 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks"] Nov 04 00:53:11.247125 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:53:11.247091 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a9a396_9047_4f41_8d49_f739f0886037.slice/crio-86f0a668f0693052731159effd2a7b54409fe6f7f581aba4c97d6d85c4ce92e7 WatchSource:0}: Error finding container 86f0a668f0693052731159effd2a7b54409fe6f7f581aba4c97d6d85c4ce92e7: Status 404 returned error can't find the container with id 86f0a668f0693052731159effd2a7b54409fe6f7f581aba4c97d6d85c4ce92e7 Nov 04 00:53:11.815307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:11.815211 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" event={"ID":"d5a9a396-9047-4f41-8d49-f739f0886037","Type":"ContainerStarted","Data":"44e0647598d1caf4dd21bcba1dcde6a4d1212d4b980ed01dfe6523d69decbe66"} Nov 04 00:53:11.815307 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:11.815271 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" event={"ID":"d5a9a396-9047-4f41-8d49-f739f0886037","Type":"ContainerStarted","Data":"86f0a668f0693052731159effd2a7b54409fe6f7f581aba4c97d6d85c4ce92e7"} Nov 04 00:53:11.835333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:11.835283 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" podStartSLOduration=1.83527091 podStartE2EDuration="1.83527091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:53:11.833240103 +0000 UTC m=+5010.655836280" watchObservedRunningTime="2026-11-04 00:53:11.83527091 +0000 UTC m=+5010.657867072" Nov 04 00:53:12.821834 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:12.821800 6208 generic.go:356] "Generic (PLEG): container finished" podID="d5a9a396-9047-4f41-8d49-f739f0886037" containerID="44e0647598d1caf4dd21bcba1dcde6a4d1212d4b980ed01dfe6523d69decbe66" exitCode=0 Nov 04 00:53:12.822288 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:12.821859 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" event={"ID":"d5a9a396-9047-4f41-8d49-f739f0886037","Type":"ContainerDied","Data":"44e0647598d1caf4dd21bcba1dcde6a4d1212d4b980ed01dfe6523d69decbe66"} Nov 04 00:53:13.991084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:13.991058 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:14.085248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.085164 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-xks-ca\") pod \"d5a9a396-9047-4f41-8d49-f739f0886037\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " Nov 04 00:53:14.085248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.085217 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a9a396-9047-4f41-8d49-f739f0886037-kube-api-access-k6tzv\" (UniqueName: \"kubernetes.io/projected/d5a9a396-9047-4f41-8d49-f739f0886037-kube-api-access-k6tzv\") pod \"d5a9a396-9047-4f41-8d49-f739f0886037\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " Nov 04 00:53:14.085452 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.085270 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-openshift-ca\") pod \"d5a9a396-9047-4f41-8d49-f739f0886037\" (UID: \"d5a9a396-9047-4f41-8d49-f739f0886037\") " Nov 04 00:53:14.085673 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.085650 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-openshift-ca" pod "d5a9a396-9047-4f41-8d49-f739f0886037" (UID: "d5a9a396-9047-4f41-8d49-f739f0886037"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:53:14.087396 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.087368 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-xks-ca" pod "d5a9a396-9047-4f41-8d49-f739f0886037" (UID: "d5a9a396-9047-4f41-8d49-f739f0886037"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:53:14.087529 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.087434 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a9a396-9047-4f41-8d49-f739f0886037-kube-api-access-k6tzv" pod "d5a9a396-9047-4f41-8d49-f739f0886037" (UID: "d5a9a396-9047-4f41-8d49-f739f0886037"). InnerVolumeSpecName "kube-api-access-k6tzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:53:14.186130 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.186086 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:53:14.186130 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.186126 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a9a396-9047-4f41-8d49-f739f0886037-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:53:14.186130 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.186137 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6tzv\" (UniqueName: \"kubernetes.io/projected/d5a9a396-9047-4f41-8d49-f739f0886037-kube-api-access-k6tzv\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:53:14.835322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.835283 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" event={"ID":"d5a9a396-9047-4f41-8d49-f739f0886037","Type":"ContainerDied","Data":"86f0a668f0693052731159effd2a7b54409fe6f7f581aba4c97d6d85c4ce92e7"} Nov 04 00:53:14.835322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.835307 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks" Nov 04 00:53:14.835550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:14.835339 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86f0a668f0693052731159effd2a7b54409fe6f7f581aba4c97d6d85c4ce92e7" Nov 04 00:53:20.446106 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.446066 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:53:20.888552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.888517 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z"] Nov 04 00:53:20.888916 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.888904 6208 cpu_manager.go:414] "containerMap: removing container" podUID="d5a9a396-9047-4f41-8d49-f739f0886037" containerName="revoke-keys" Nov 04 00:53:20.888968 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.888917 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a9a396-9047-4f41-8d49-f739f0886037" containerName="revoke-keys" Nov 04 00:53:20.889017 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.888979 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a9a396-9047-4f41-8d49-f739f0886037" containerName="revoke-keys" Nov 04 00:53:20.892459 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.892436 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:20.902239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.902213 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z"] Nov 04 00:53:20.903863 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.903801 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:53:20.903992 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.903973 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:53:20.904130 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.904108 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:53:20.904205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:20.904190 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:53:21.048474 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.048434 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-hv27z\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.048650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.048485 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-hv27z\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.048650 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.048610 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9fm\" (UniqueName: \"kubernetes.io/projected/74ec63b4-767f-415d-8bc6-ac95870d8c9c-kube-api-access-tj9fm\") pod \"maas-api-revoke-sub-5a29efb3-hv27z\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.149215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.149123 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9fm\" (UniqueName: \"kubernetes.io/projected/74ec63b4-767f-415d-8bc6-ac95870d8c9c-kube-api-access-tj9fm\") pod \"maas-api-revoke-sub-5a29efb3-hv27z\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.149422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.149234 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-hv27z\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.149422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.149275 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-hv27z\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.149889 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.149866 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-hv27z\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.151628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.151610 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-hv27z\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.162240 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.162210 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9fm\" (UniqueName: \"kubernetes.io/projected/74ec63b4-767f-415d-8bc6-ac95870d8c9c-kube-api-access-tj9fm\") pod \"maas-api-revoke-sub-5a29efb3-hv27z\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.207022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.206989 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:21.482287 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.482257 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z"] Nov 04 00:53:21.484077 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:53:21.484050 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ec63b4_767f_415d_8bc6_ac95870d8c9c.slice/crio-820a9598aae4f5466e1078e79a3efed1f9af772fca54438a14db71ab02dfc8e8 WatchSource:0}: Error finding container 820a9598aae4f5466e1078e79a3efed1f9af772fca54438a14db71ab02dfc8e8: Status 404 returned error can't find the container with id 820a9598aae4f5466e1078e79a3efed1f9af772fca54438a14db71ab02dfc8e8 Nov 04 00:53:21.874706 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.874663 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" event={"ID":"74ec63b4-767f-415d-8bc6-ac95870d8c9c","Type":"ContainerStarted","Data":"fce657c0514b1dd07917e2e357a57481678b75e20d8e66926c3167d1e30b0625"} Nov 04 00:53:21.874919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.874726 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" event={"ID":"74ec63b4-767f-415d-8bc6-ac95870d8c9c","Type":"ContainerStarted","Data":"820a9598aae4f5466e1078e79a3efed1f9af772fca54438a14db71ab02dfc8e8"} Nov 04 00:53:21.894062 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:21.893996 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" podStartSLOduration=1.893980129 podStartE2EDuration="1.893980129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:53:21.8929443 +0000 UTC m=+5020.715540464" watchObservedRunningTime="2026-11-04 00:53:21.893980129 +0000 UTC m=+5020.716576336" Nov 04 00:53:22.904403 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:22.904352 6208 generic.go:356] "Generic (PLEG): container finished" podID="74ec63b4-767f-415d-8bc6-ac95870d8c9c" containerID="fce657c0514b1dd07917e2e357a57481678b75e20d8e66926c3167d1e30b0625" exitCode=0 Nov 04 00:53:22.905153 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:22.905124 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" event={"ID":"74ec63b4-767f-415d-8bc6-ac95870d8c9c","Type":"ContainerDied","Data":"fce657c0514b1dd07917e2e357a57481678b75e20d8e66926c3167d1e30b0625"} Nov 04 00:53:24.244899 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.244875 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:24.430000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.429018 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ec63b4-767f-415d-8bc6-ac95870d8c9c-kube-api-access-tj9fm\" (UniqueName: \"kubernetes.io/projected/74ec63b4-767f-415d-8bc6-ac95870d8c9c-kube-api-access-tj9fm\") pod \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " Nov 04 00:53:24.430000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.429185 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-openshift-ca\") pod \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " Nov 04 00:53:24.430000 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.429248 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-xks-ca\") pod \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\" (UID: \"74ec63b4-767f-415d-8bc6-ac95870d8c9c\") " Nov 04 00:53:24.431116 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.431086 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-openshift-ca" pod "74ec63b4-767f-415d-8bc6-ac95870d8c9c" (UID: "74ec63b4-767f-415d-8bc6-ac95870d8c9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:53:24.432373 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.432346 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-xks-ca" pod "74ec63b4-767f-415d-8bc6-ac95870d8c9c" (UID: "74ec63b4-767f-415d-8bc6-ac95870d8c9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:53:24.433506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.433475 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ec63b4-767f-415d-8bc6-ac95870d8c9c-kube-api-access-tj9fm" pod "74ec63b4-767f-415d-8bc6-ac95870d8c9c" (UID: "74ec63b4-767f-415d-8bc6-ac95870d8c9c"). InnerVolumeSpecName "kube-api-access-tj9fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:53:24.530409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.530301 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj9fm\" (UniqueName: \"kubernetes.io/projected/74ec63b4-767f-415d-8bc6-ac95870d8c9c-kube-api-access-tj9fm\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:53:24.530409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.530362 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:53:24.530409 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.530381 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ec63b4-767f-415d-8bc6-ac95870d8c9c-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:53:24.894325 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.894240 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:53:24.929836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.929808 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" Nov 04 00:53:24.929836 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.929817 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z" event={"ID":"74ec63b4-767f-415d-8bc6-ac95870d8c9c","Type":"ContainerDied","Data":"820a9598aae4f5466e1078e79a3efed1f9af772fca54438a14db71ab02dfc8e8"} Nov 04 00:53:24.930060 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:53:24.929864 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="820a9598aae4f5466e1078e79a3efed1f9af772fca54438a14db71ab02dfc8e8" Nov 04 00:53:35.707172 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:53:35.707135 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:54:05.702792 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:54:05.702751 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:54:07.326144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.326097 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45"] Nov 04 00:54:07.327357 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.327336 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4p45"] Nov 04 00:54:07.346563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.346533 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4"] Nov 04 00:54:07.346959 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.346944 6208 cpu_manager.go:414] "containerMap: removing container" podUID="74ec63b4-767f-415d-8bc6-ac95870d8c9c" containerName="revoke-keys" Nov 04 00:54:07.347009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.346962 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ec63b4-767f-415d-8bc6-ac95870d8c9c" containerName="revoke-keys" Nov 04 00:54:07.347070 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.347050 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ec63b4-767f-415d-8bc6-ac95870d8c9c" containerName="revoke-keys" Nov 04 00:54:07.350315 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.350292 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.356563 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.356539 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:54:07.356729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.356708 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:54:07.361561 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.361532 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4"] Nov 04 00:54:07.431009 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.430978 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:54:07.431199 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.431077 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:54:07.540710 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.540672 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgdt4\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.540893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.540727 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8pb\" (UniqueName: \"kubernetes.io/projected/35713749-fbab-402c-b91b-431cade2c8eb-kube-api-access-qn8pb\") pod \"maas-api-revoke-sub-3d4156f1-cgdt4\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.540893 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.540862 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgdt4\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.641377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.641295 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgdt4\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.641377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.641342 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8pb\" (UniqueName: \"kubernetes.io/projected/35713749-fbab-402c-b91b-431cade2c8eb-kube-api-access-qn8pb\") pod \"maas-api-revoke-sub-3d4156f1-cgdt4\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.641578 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.641397 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgdt4\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.642049 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.642019 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgdt4\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.643828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.643806 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgdt4\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.653885 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.653859 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8pb\" (UniqueName: \"kubernetes.io/projected/35713749-fbab-402c-b91b-431cade2c8eb-kube-api-access-qn8pb\") pod \"maas-api-revoke-sub-3d4156f1-cgdt4\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.665903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.665882 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:07.946433 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:07.946401 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4"] Nov 04 00:54:07.948085 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:54:07.948058 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35713749_fbab_402c_b91b_431cade2c8eb.slice/crio-c0a055866692f5d693083964b47fec753144eaf6b35275da00fe93267f86a9d9 WatchSource:0}: Error finding container c0a055866692f5d693083964b47fec753144eaf6b35275da00fe93267f86a9d9: Status 404 returned error can't find the container with id c0a055866692f5d693083964b47fec753144eaf6b35275da00fe93267f86a9d9 Nov 04 00:54:08.174853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:08.174821 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" event={"ID":"35713749-fbab-402c-b91b-431cade2c8eb","Type":"ContainerStarted","Data":"c0a055866692f5d693083964b47fec753144eaf6b35275da00fe93267f86a9d9"} Nov 04 00:54:09.088951 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:09.088875 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:09.181577 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:09.181540 6208 generic.go:356] "Generic (PLEG): container finished" podID="35713749-fbab-402c-b91b-431cade2c8eb" containerID="88774b9c5691b55f7d87fd4b9f46c333c6cf65afe16aceb45bf90bfb0463d136" exitCode=0 Nov 04 00:54:09.181774 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:09.181639 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" event={"ID":"35713749-fbab-402c-b91b-431cade2c8eb","Type":"ContainerDied","Data":"88774b9c5691b55f7d87fd4b9f46c333c6cf65afe16aceb45bf90bfb0463d136"} Nov 04 00:54:09.292740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:09.292709 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8ef494-5763-4ec9-b1c7-2918f560b943" path="/var/lib/kubelet/pods/6b8ef494-5763-4ec9-b1c7-2918f560b943/volumes" Nov 04 00:54:10.320473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.320449 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:10.363677 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.363643 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-xks-ca\") pod \"35713749-fbab-402c-b91b-431cade2c8eb\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " Nov 04 00:54:10.363822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.363788 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35713749-fbab-402c-b91b-431cade2c8eb-kube-api-access-qn8pb\" (UniqueName: \"kubernetes.io/projected/35713749-fbab-402c-b91b-431cade2c8eb-kube-api-access-qn8pb\") pod \"35713749-fbab-402c-b91b-431cade2c8eb\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " Nov 04 00:54:10.363822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.363816 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-openshift-ca\") pod \"35713749-fbab-402c-b91b-431cade2c8eb\" (UID: \"35713749-fbab-402c-b91b-431cade2c8eb\") " Nov 04 00:54:10.364299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.364266 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-openshift-ca" pod "35713749-fbab-402c-b91b-431cade2c8eb" (UID: "35713749-fbab-402c-b91b-431cade2c8eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:54:10.365705 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.365673 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-xks-ca" pod "35713749-fbab-402c-b91b-431cade2c8eb" (UID: "35713749-fbab-402c-b91b-431cade2c8eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:54:10.365896 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.365875 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35713749-fbab-402c-b91b-431cade2c8eb-kube-api-access-qn8pb" pod "35713749-fbab-402c-b91b-431cade2c8eb" (UID: "35713749-fbab-402c-b91b-431cade2c8eb"). InnerVolumeSpecName "kube-api-access-qn8pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:54:10.465248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.465212 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8pb\" (UniqueName: \"kubernetes.io/projected/35713749-fbab-402c-b91b-431cade2c8eb-kube-api-access-qn8pb\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:54:10.465248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.465245 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:54:10.465443 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:10.465261 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35713749-fbab-402c-b91b-431cade2c8eb-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:54:11.194824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:11.194779 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" event={"ID":"35713749-fbab-402c-b91b-431cade2c8eb","Type":"ContainerDied","Data":"c0a055866692f5d693083964b47fec753144eaf6b35275da00fe93267f86a9d9"} Nov 04 00:54:11.194824 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:11.194809 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4" Nov 04 00:54:11.195082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:11.194835 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0a055866692f5d693083964b47fec753144eaf6b35275da00fe93267f86a9d9" Nov 04 00:54:20.536911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:20.536870 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:24.957617 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:24.957578 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:32.533282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:32.533245 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:32.939216 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:32.939132 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl"] Nov 04 00:54:32.939550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:32.939529 6208 cpu_manager.go:414] "containerMap: removing container" podUID="35713749-fbab-402c-b91b-431cade2c8eb" containerName="revoke-keys" Nov 04 00:54:32.939550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:32.939547 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35713749-fbab-402c-b91b-431cade2c8eb" containerName="revoke-keys" Nov 04 00:54:32.939760 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:32.939637 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="35713749-fbab-402c-b91b-431cade2c8eb" containerName="revoke-keys" Nov 04 00:54:32.942886 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:32.942864 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:32.948552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:32.948526 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b34223-openshift-default-dockercfg-j22g8\"" Nov 04 00:54:32.957347 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:32.957323 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl"] Nov 04 00:54:32.981696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:32.981667 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:33.035532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.035498 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k"] Nov 04 00:54:33.043128 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.043099 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-dhh5k"] Nov 04 00:54:33.106157 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.106124 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrws7\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-kube-api-access-wrws7\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.106323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.106184 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-token\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.106323 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.106310 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-envoy\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.106451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.106342 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-socket\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.106451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.106367 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b989dd-410a-4fb1-91b8-8af60c1186bd-istiod-ca-cert\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.106451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.106428 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-data\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.106451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.106449 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-credential-socket\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.106620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.106467 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-podinfo\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.106620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.106484 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-certs\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.138767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.138732 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:33.207456 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.207427 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrws7\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-kube-api-access-wrws7\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.207646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.207483 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-token\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.207646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.207539 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-envoy\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.207646 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.207562 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-socket\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.207793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.207692 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b989dd-410a-4fb1-91b8-8af60c1186bd-istiod-ca-cert\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.207793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.207755 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-data\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.207793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.207787 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-credential-socket\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.208248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.207804 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-podinfo\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.208248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.207832 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-certs\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.208248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.208090 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-socket\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.208248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.208188 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-data\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.208436 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.208255 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-credential-socket\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.208436 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.208274 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-certs\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.208436 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.208367 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b989dd-410a-4fb1-91b8-8af60c1186bd-istiod-ca-cert\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.210172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.210150 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-envoy\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.210334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.210315 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-podinfo\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.219956 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.219938 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-token\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.220363 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.220341 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrws7\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-kube-api-access-wrws7\") pod \"e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.257396 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.257369 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:33.294795 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.294765 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89202ebf-5bce-4078-ad92-9bd176d929f7" path="/var/lib/kubelet/pods/89202ebf-5bce-4078-ad92-9bd176d929f7/volumes" Nov 04 00:54:33.542535 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.542499 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl"] Nov 04 00:54:33.544977 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:54:33.544941 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b989dd_410a_4fb1_91b8_8af60c1186bd.slice/crio-b369f00f7257f39906bd2f735df5fb6850baa69588da5cee91c27536ae0b471b WatchSource:0}: Error finding container b369f00f7257f39906bd2f735df5fb6850baa69588da5cee91c27536ae0b471b: Status 404 returned error can't find the container with id b369f00f7257f39906bd2f735df5fb6850baa69588da5cee91c27536ae0b471b Nov 04 00:54:33.547242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.547198 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:54:33.547333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.547285 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:54:33.547378 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:33.547329 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:54:34.359842 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:34.359801 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" event={"ID":"50b989dd-410a-4fb1-91b8-8af60c1186bd","Type":"ContainerStarted","Data":"30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf"} Nov 04 00:54:34.360085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:34.359859 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" event={"ID":"50b989dd-410a-4fb1-91b8-8af60c1186bd","Type":"ContainerStarted","Data":"b369f00f7257f39906bd2f735df5fb6850baa69588da5cee91c27536ae0b471b"} Nov 04 00:54:35.257941 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:35.257906 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:35.263107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:35.263078 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:35.288950 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:35.288889 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" podStartSLOduration=3.288871977 podStartE2EDuration="3.288871977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:54:34.386583019 +0000 UTC m=+5093.209179180" watchObservedRunningTime="2026-11-04 00:54:35.288871977 +0000 UTC m=+5094.111468139" Nov 04 00:54:35.365750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:35.365719 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:35.366621 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:35.366602 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:35.703823 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:54:35.703739 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:54:36.692230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:36.692190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:36.789684 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:36.789620 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:37.882177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.882139 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:37.899562 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.899528 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph"] Nov 04 00:54:37.903610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.903584 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:37.914228 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.914194 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:54:37.915815 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.915571 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:54:37.916215 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.916192 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq"] Nov 04 00:54:37.920389 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.920369 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph"] Nov 04 00:54:37.920541 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.920500 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:37.928770 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.928745 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq"] Nov 04 00:54:37.953444 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.953386 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b34223-serving-cert\"" Nov 04 00:54:37.953665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.953644 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 00:54:37.953768 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.953702 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xflt\"" Nov 04 00:54:37.987420 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:37.987386 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:38.012516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.012490 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b34223\"" Nov 04 00:54:38.057549 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.057512 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpgk\" (UniqueName: \"kubernetes.io/projected/f8c940b8-354d-44a5-a0fe-6738d544fb19-kube-api-access-xlpgk\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq\" (UID: \"f8c940b8-354d-44a5-a0fe-6738d544fb19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:38.057714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.057595 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.057714 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.057660 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83108053-a222-4664-a2d0-591c077a9ca3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.057793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.057715 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmpz\" (UniqueName: \"kubernetes.io/projected/83108053-a222-4664-a2d0-591c077a9ca3-kube-api-access-bbmpz\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.057845 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.057824 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.057883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.057864 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c940b8-354d-44a5-a0fe-6738d544fb19-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq\" (UID: \"f8c940b8-354d-44a5-a0fe-6738d544fb19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:38.104405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.104357 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:38.159559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.159440 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmpz\" (UniqueName: \"kubernetes.io/projected/83108053-a222-4664-a2d0-591c077a9ca3-kube-api-access-bbmpz\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.159559 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.159542 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.159911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.159596 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c940b8-354d-44a5-a0fe-6738d544fb19-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq\" (UID: \"f8c940b8-354d-44a5-a0fe-6738d544fb19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:38.159911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.159692 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpgk\" (UniqueName: \"kubernetes.io/projected/f8c940b8-354d-44a5-a0fe-6738d544fb19-kube-api-access-xlpgk\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq\" (UID: \"f8c940b8-354d-44a5-a0fe-6738d544fb19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:38.159911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.159722 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.159911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.159762 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83108053-a222-4664-a2d0-591c077a9ca3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.161388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.161354 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c940b8-354d-44a5-a0fe-6738d544fb19-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq\" (UID: \"f8c940b8-354d-44a5-a0fe-6738d544fb19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:38.161741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.161714 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83108053-a222-4664-a2d0-591c077a9ca3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.164115 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.164026 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.164788 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.164762 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.175278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.174960 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpgk\" (UniqueName: \"kubernetes.io/projected/f8c940b8-354d-44a5-a0fe-6738d544fb19-kube-api-access-xlpgk\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq\" (UID: \"f8c940b8-354d-44a5-a0fe-6738d544fb19\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:38.175278 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.175156 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmpz\" (UniqueName: \"kubernetes.io/projected/83108053-a222-4664-a2d0-591c077a9ca3-kube-api-access-bbmpz\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.225700 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.225668 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:38.238354 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.238334 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:38.516294 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.516269 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph"] Nov 04 00:54:38.517626 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:54:38.517598 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83108053_a222_4664_a2d0_591c077a9ca3.slice/crio-27449622f4062f1f4084f6c959dc09d2357da6e52d1d2d7c74a188dbc0befc0b WatchSource:0}: Error finding container 27449622f4062f1f4084f6c959dc09d2357da6e52d1d2d7c74a188dbc0befc0b: Status 404 returned error can't find the container with id 27449622f4062f1f4084f6c959dc09d2357da6e52d1d2d7c74a188dbc0befc0b Nov 04 00:54:38.519137 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.519117 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:54:38.531065 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.531023 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq"] Nov 04 00:54:38.532395 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:54:38.532372 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c940b8_354d_44a5_a0fe_6738d544fb19.slice/crio-b3e59c0467d8c42817e2428f983f885085d2127b7c2f7b0bf5811f354bb3d67e WatchSource:0}: Error finding container b3e59c0467d8c42817e2428f983f885085d2127b7c2f7b0bf5811f354bb3d67e: Status 404 returned error can't find the container with id b3e59c0467d8c42817e2428f983f885085d2127b7c2f7b0bf5811f354bb3d67e Nov 04 00:54:38.835815 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:38.835708 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:39.394090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:39.393984 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" event={"ID":"f8c940b8-354d-44a5-a0fe-6738d544fb19","Type":"ContainerStarted","Data":"0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa"} Nov 04 00:54:39.394090 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:39.394067 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" event={"ID":"f8c940b8-354d-44a5-a0fe-6738d544fb19","Type":"ContainerStarted","Data":"b3e59c0467d8c42817e2428f983f885085d2127b7c2f7b0bf5811f354bb3d67e"} Nov 04 00:54:39.394596 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:39.394234 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:39.395827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:39.395800 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" event={"ID":"83108053-a222-4664-a2d0-591c077a9ca3","Type":"ContainerStarted","Data":"fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a"} Nov 04 00:54:39.395937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:39.395847 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:39.395937 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:39.395866 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" event={"ID":"83108053-a222-4664-a2d0-591c077a9ca3","Type":"ContainerStarted","Data":"27449622f4062f1f4084f6c959dc09d2357da6e52d1d2d7c74a188dbc0befc0b"} Nov 04 00:54:39.414587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:39.414538 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" podStartSLOduration=2.414526447 podStartE2EDuration="2.414526447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:54:39.412242305 +0000 UTC m=+5098.234838465" watchObservedRunningTime="2026-11-04 00:54:39.414526447 +0000 UTC m=+5098.237122608" Nov 04 00:54:39.432894 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:39.432852 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" podStartSLOduration=1.857882018 podStartE2EDuration="2.432841382s" totalImagesPullingTime="574.959364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:54:39.431772194 +0000 UTC m=+5098.254368357" watchObservedRunningTime="2026-11-04 00:54:39.432841382 +0000 UTC m=+5098.255437574" Nov 04 00:54:39.629949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:39.629905 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:40.450488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:40.450446 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:41.331583 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:41.331530 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:42.030238 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:42.030194 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:42.746797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:42.746758 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:43.297089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:43.297055 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:54:43.300926 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:43.300897 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:54:43.437155 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:43.437111 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:44.248464 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:44.248429 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:44.935903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:44.935864 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:45.406822 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:45.406790 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:54:45.679135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:45.679009 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:46.501302 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:46.501254 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:47.335922 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:47.335879 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:48.048848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:48.048803 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:48.834800 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:48.834761 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:49.733782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:49.733745 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:50.405190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:50.405153 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:54:50.453912 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:50.453869 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:50.669572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:50.669089 6208 patch_prober.go:29] interesting pod/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": read tcp 10.133.0.2:34250->10.133.0.235:15021: read: connection reset by peer" start-of-body= Nov 04 00:54:50.669572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:50.669160 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": read tcp 10.133.0.2:34250->10.133.0.235:15021: read: connection reset by peer" Nov 04 00:54:51.230483 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:51.230447 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:51.470056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:51.469999 6208 generic.go:356] "Generic (PLEG): container finished" podID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerID="30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf" exitCode=0 Nov 04 00:54:51.470244 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:51.470077 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" event={"ID":"50b989dd-410a-4fb1-91b8-8af60c1186bd","Type":"ContainerDied","Data":"30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf"} Nov 04 00:54:51.470500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:51.470485 6208 scope.go:122] "RemoveContainer" containerID="30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf" Nov 04 00:54:51.471692 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:51.471665 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:54:51.471763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:51.471736 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:54:51.471801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:51.471768 6208 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 00:54:52.019061 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:52.019014 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:52.489334 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:52.489296 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" event={"ID":"50b989dd-410a-4fb1-91b8-8af60c1186bd","Type":"ContainerStarted","Data":"1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad"} Nov 04 00:54:52.742763 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:52.742669 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:53.028164 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:53.028075 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c"] Nov 04 00:54:53.030475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:53.030441 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnp5c"] Nov 04 00:54:53.270066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:53.269424 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:53.270066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:53.269590 6208 patch_prober.go:29] interesting pod/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 04 00:54:53.270066 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:53.269641 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 04 00:54:53.292978 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:53.292905 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016f274c-500d-4ccb-9749-205f1b4c1f48" path="/var/lib/kubelet/pods/016f274c-500d-4ccb-9749-205f1b4c1f48/volumes" Nov 04 00:54:53.520571 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:53.520538 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:54.263057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:54.261441 6208 patch_prober.go:29] interesting pod/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 04 00:54:54.263057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:54.261514 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 04 00:54:54.267935 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:54.264012 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:55.041645 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:55.041611 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:55.274056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:55.268406 6208 patch_prober.go:29] interesting pod/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 04 00:54:55.274056 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:55.268472 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 04 00:54:55.831225 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:55.831190 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:56.267476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:56.267434 6208 patch_prober.go:29] interesting pod/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 04 00:54:56.268108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:56.268071 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 04 00:54:56.579141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:56.579022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:57.272069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:57.266479 6208 patch_prober.go:29] interesting pod/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Nov 04 00:54:57.272069 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:57.266544 6208 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Nov 04 00:54:57.335655 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:57.335619 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:58.070959 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:58.070912 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:58.261560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:58.261524 6208 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:58.261934 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:58.261909 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:58.262634 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:58.262614 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:54:58.833721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:58.833680 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:54:59.645990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:54:59.645949 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:00.474820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:00.474788 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:01.140270 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:01.140234 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:01.756588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:01.756548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:01.933786 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:01.933750 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:02.412743 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:02.412707 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:02.516949 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:02.516885 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:02.623187 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:02.623139 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:02.743414 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:02.743378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:03.644333 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:03.644271 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:04.455969 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:04.455911 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:04.550846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:04.550807 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:05.253818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:05.253773 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:05.705159 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:55:05.705067 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:55:06.044765 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:06.044730 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:07.056871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:07.056809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:07.845930 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:07.845878 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:08.640093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:08.640022 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:09.140887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:09.140851 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:09.549135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:09.549089 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:10.031906 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:10.031869 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2"] Nov 04 00:55:10.034168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:10.034139 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-b6hp2"] Nov 04 00:55:10.324064 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:10.323940 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:11.035183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:11.035146 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:11.300961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:11.300869 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f44bc50-0641-41f7-9288-f930dd6b3ca6" path="/var/lib/kubelet/pods/1f44bc50-0641-41f7-9288-f930dd6b3ca6/volumes" Nov 04 00:55:11.818532 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:11.818495 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:12.537931 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:12.537896 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:13.230058 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:13.229987 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:13.946506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:13.946470 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:14.714250 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:14.714219 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:15.546497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:15.546456 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:16.409752 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:16.409717 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:17.126854 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:17.126815 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:17.908405 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:17.908354 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:18.646990 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:18.646952 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:19.427772 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:19.427738 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:20.143107 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:20.143069 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:20.614365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:20.614332 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:21.025251 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:21.025211 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:21.027256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:21.027223 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p"] Nov 04 00:55:21.030965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:21.030938 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jbr4p"] Nov 04 00:55:21.293598 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:21.293509 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c94aa54-0479-48f3-b6a9-8cb4fe47150c" path="/var/lib/kubelet/pods/4c94aa54-0479-48f3-b6a9-8cb4fe47150c/volumes" Nov 04 00:55:21.836570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:21.836531 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:22.730522 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:22.730489 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:23.452942 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:23.452907 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:24.232253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:24.232219 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:25.010478 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:25.010443 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:25.111588 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:25.111555 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:25.729919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:25.729885 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:26.443433 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:26.443401 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:27.247169 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:27.247132 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:28.032555 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:28.032518 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:28.737970 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:28.737929 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:29.521569 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:29.521536 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:30.346828 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:30.346786 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:31.140514 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:31.140474 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:31.934425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:31.934372 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:32.738695 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:32.738665 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:33.633082 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:33.633026 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:34.422210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:34.422174 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:35.143430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:35.143391 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:35.705375 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:55:35.705336 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:55:36.023665 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:36.023554 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:36.833985 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:36.833948 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:37.623099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:37.623065 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:38.335356 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:38.335320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:39.115777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:39.115742 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:39.836302 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:39.836260 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:40.568851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:40.568811 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:41.327870 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:41.327836 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:42.052314 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:42.052267 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:42.920783 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:42.920741 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:43.645925 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:43.645885 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:44.350495 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:44.350461 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:45.142008 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:45.141970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:45.903475 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:45.903435 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:46.638473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:46.638436 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:47.307601 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:47.307565 6208 scope.go:122] "RemoveContainer" containerID="ce5163dd655a283603cf4419ec97f63f333fdff747e95b434b2f8feed97011ed" Nov 04 00:55:47.431609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:47.431578 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:48.027664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:48.027631 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf"] Nov 04 00:55:48.031145 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:48.031117 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-x9whf"] Nov 04 00:55:48.238361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:48.238320 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:49.271729 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:49.271688 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:49.293284 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:49.293255 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c402cae-28ff-4bfc-a791-c62fc095ee20" path="/var/lib/kubelet/pods/5c402cae-28ff-4bfc-a791-c62fc095ee20/volumes" Nov 04 00:55:50.041346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:50.041313 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:50.819808 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:50.819771 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:51.547425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:51.547378 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:52.350523 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:52.350485 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:53.140887 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:53.140848 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:53.849537 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:53.849504 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:54.642749 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:54.642708 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:54.944513 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:54.944430 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:55.439150 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:55.439114 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:55.541397 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:55.541364 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:56.235201 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:56.235166 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:57.044359 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:57.044322 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:57.931299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:57.931266 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:58.062905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:58.062862 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k"] Nov 04 00:55:58.065345 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:58.065321 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-prm4k"] Nov 04 00:55:58.829357 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:58.829322 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:55:59.292580 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:59.292545 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82cf639-f285-4285-a5bb-8de2e947e3bb" path="/var/lib/kubelet/pods/c82cf639-f285-4285-a5bb-8de2e947e3bb/volumes" Nov 04 00:55:59.630013 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:55:59.629899 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:00.007944 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:00.007905 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:00.409910 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:00.409827 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:01.232644 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:01.232608 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:01.510055 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:01.509951 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq"] Nov 04 00:56:01.567615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:01.567577 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:01.667344 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:01.667301 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:02.142496 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:02.142451 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:02.799739 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:02.799704 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:02.804210 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:02.804180 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph"] Nov 04 00:56:02.804406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:02.804381 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" podUID="83108053-a222-4664-a2d0-591c077a9ca3" containerName="maas-api" containerID="cri-o://fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a" gracePeriod=30 Nov 04 00:56:02.906953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:02.906918 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:03.255002 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.254978 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:56:03.398921 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.398825 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-metrics-certs\") pod \"83108053-a222-4664-a2d0-591c077a9ca3\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " Nov 04 00:56:03.399133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.398990 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83108053-a222-4664-a2d0-591c077a9ca3-kube-api-access-bbmpz\" (UniqueName: \"kubernetes.io/projected/83108053-a222-4664-a2d0-591c077a9ca3-kube-api-access-bbmpz\") pod \"83108053-a222-4664-a2d0-591c077a9ca3\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " Nov 04 00:56:03.399133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.399023 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-maas-api-tls\") pod \"83108053-a222-4664-a2d0-591c077a9ca3\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " Nov 04 00:56:03.399133 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.399122 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83108053-a222-4664-a2d0-591c077a9ca3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83108053-a222-4664-a2d0-591c077a9ca3-openshift-service-ca\") pod \"83108053-a222-4664-a2d0-591c077a9ca3\" (UID: \"83108053-a222-4664-a2d0-591c077a9ca3\") " Nov 04 00:56:03.399506 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.399477 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83108053-a222-4664-a2d0-591c077a9ca3-openshift-service-ca" pod "83108053-a222-4664-a2d0-591c077a9ca3" (UID: "83108053-a222-4664-a2d0-591c077a9ca3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:56:03.401190 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.401162 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83108053-a222-4664-a2d0-591c077a9ca3-kube-api-access-bbmpz" pod "83108053-a222-4664-a2d0-591c077a9ca3" (UID: "83108053-a222-4664-a2d0-591c077a9ca3"). InnerVolumeSpecName "kube-api-access-bbmpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:56:03.401306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.401162 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-maas-api-tls" pod "83108053-a222-4664-a2d0-591c077a9ca3" (UID: "83108053-a222-4664-a2d0-591c077a9ca3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:56:03.401306 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.401208 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-metrics-certs" pod "83108053-a222-4664-a2d0-591c077a9ca3" (UID: "83108053-a222-4664-a2d0-591c077a9ca3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:56:03.500542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.500500 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:03.500542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.500536 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbmpz\" (UniqueName: \"kubernetes.io/projected/83108053-a222-4664-a2d0-591c077a9ca3-kube-api-access-bbmpz\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:03.500542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.500546 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83108053-a222-4664-a2d0-591c077a9ca3-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:03.500791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.500555 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83108053-a222-4664-a2d0-591c077a9ca3-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:03.904158 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.904116 6208 generic.go:356] "Generic (PLEG): container finished" podID="83108053-a222-4664-a2d0-591c077a9ca3" containerID="fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a" exitCode=0 Nov 04 00:56:03.904631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.904171 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" Nov 04 00:56:03.904631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.904174 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" event={"ID":"83108053-a222-4664-a2d0-591c077a9ca3","Type":"ContainerDied","Data":"fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a"} Nov 04 00:56:03.904631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.904228 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph" event={"ID":"83108053-a222-4664-a2d0-591c077a9ca3","Type":"ContainerDied","Data":"27449622f4062f1f4084f6c959dc09d2357da6e52d1d2d7c74a188dbc0befc0b"} Nov 04 00:56:03.904631 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.904258 6208 scope.go:122] "RemoveContainer" containerID="fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a" Nov 04 00:56:03.914265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.914244 6208 scope.go:122] "RemoveContainer" containerID="fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a" Nov 04 00:56:03.914554 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:56:03.914531 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a\": container with ID starting with fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a not found: ID does not exist" containerID="fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a" Nov 04 00:56:03.914605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.914565 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a"} err="failed to get container status \"fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a\": rpc error: code = NotFound desc = could not find container \"fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a\": container with ID starting with fd1587d7f43feabe0ba97112edc547cc13b8f87cfa0e9cc56e3a12b25777bb8a not found: ID does not exist" Nov 04 00:56:03.927324 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.927293 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph"] Nov 04 00:56:03.930748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:03.930721 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-bzmph"] Nov 04 00:56:05.293820 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:05.293784 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83108053-a222-4664-a2d0-591c077a9ca3" path="/var/lib/kubelet/pods/83108053-a222-4664-a2d0-591c077a9ca3/volumes" Nov 04 00:56:05.703839 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:56:05.703757 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:56:06.510756 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:06.510695 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" podUID="f8c940b8-354d-44a5-a0fe-6738d544fb19" containerName="payload-processing" containerID="cri-o://0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa" gracePeriod=25 Nov 04 00:56:08.795171 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:08.795114 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:08.891298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:08.891259 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:09.208269 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:09.208231 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:10.403900 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:10.403858 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" podUID="f8c940b8-354d-44a5-a0fe-6738d544fb19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:56:19.026834 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:19.026796 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n"] Nov 04 00:56:19.032057 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:19.032000 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-k758n"] Nov 04 00:56:19.293319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:19.293237 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54de174b-a2c7-4668-8ba9-355a1783b1ef" path="/var/lib/kubelet/pods/54de174b-a2c7-4668-8ba9-355a1783b1ef/volumes" Nov 04 00:56:20.403408 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:20.403372 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" podUID="f8c940b8-354d-44a5-a0fe-6738d544fb19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 00:56:20.678905 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:20.678808 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:25.063967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:25.063930 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:26.759707 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:26.759682 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:56:26.827417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:26.827328 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c940b8-354d-44a5-a0fe-6738d544fb19-kube-api-access-xlpgk\" (UniqueName: \"kubernetes.io/projected/f8c940b8-354d-44a5-a0fe-6738d544fb19-kube-api-access-xlpgk\") pod \"f8c940b8-354d-44a5-a0fe-6738d544fb19\" (UID: \"f8c940b8-354d-44a5-a0fe-6738d544fb19\") " Nov 04 00:56:26.827417 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:26.827389 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c940b8-354d-44a5-a0fe-6738d544fb19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c940b8-354d-44a5-a0fe-6738d544fb19-plugins-config-volume\") pod \"f8c940b8-354d-44a5-a0fe-6738d544fb19\" (UID: \"f8c940b8-354d-44a5-a0fe-6738d544fb19\") " Nov 04 00:56:26.827813 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:26.827782 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c940b8-354d-44a5-a0fe-6738d544fb19-plugins-config-volume" pod "f8c940b8-354d-44a5-a0fe-6738d544fb19" (UID: "f8c940b8-354d-44a5-a0fe-6738d544fb19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:56:26.829553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:26.829527 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c940b8-354d-44a5-a0fe-6738d544fb19-kube-api-access-xlpgk" pod "f8c940b8-354d-44a5-a0fe-6738d544fb19" (UID: "f8c940b8-354d-44a5-a0fe-6738d544fb19"). InnerVolumeSpecName "kube-api-access-xlpgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:56:26.928542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:26.928495 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlpgk\" (UniqueName: \"kubernetes.io/projected/f8c940b8-354d-44a5-a0fe-6738d544fb19-kube-api-access-xlpgk\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:26.928542 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:26.928547 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8c940b8-354d-44a5-a0fe-6738d544fb19-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:27.035095 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.035026 6208 generic.go:356] "Generic (PLEG): container finished" podID="f8c940b8-354d-44a5-a0fe-6738d544fb19" containerID="0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa" exitCode=0 Nov 04 00:56:27.035299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.035108 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" Nov 04 00:56:27.035299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.035064 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" event={"ID":"f8c940b8-354d-44a5-a0fe-6738d544fb19","Type":"ContainerDied","Data":"0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa"} Nov 04 00:56:27.035299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.035207 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq" event={"ID":"f8c940b8-354d-44a5-a0fe-6738d544fb19","Type":"ContainerDied","Data":"b3e59c0467d8c42817e2428f983f885085d2127b7c2f7b0bf5811f354bb3d67e"} Nov 04 00:56:27.035299 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.035242 6208 scope.go:122] "RemoveContainer" containerID="0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa" Nov 04 00:56:27.047859 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.047829 6208 scope.go:122] "RemoveContainer" containerID="0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa" Nov 04 00:56:27.048202 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:56:27.048172 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa\": container with ID starting with 0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa not found: ID does not exist" containerID="0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa" Nov 04 00:56:27.048319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.048212 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa"} err="failed to get container status \"0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa\": rpc error: code = NotFound desc = could not find container \"0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa\": container with ID starting with 0a70496ec29669dd5fe462cc1ce023a8c303b6237e8ce55ac875b338e0679cfa not found: ID does not exist" Nov 04 00:56:27.061715 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.061689 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq"] Nov 04 00:56:27.065282 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.065260 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dz6hgq"] Nov 04 00:56:27.293553 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:27.293517 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c940b8-354d-44a5-a0fe-6738d544fb19" path="/var/lib/kubelet/pods/f8c940b8-354d-44a5-a0fe-6738d544fb19/volumes" Nov 04 00:56:28.947026 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:28.946982 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:33.159685 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:33.159648 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:33.348099 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:33.348063 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:34.329892 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.329850 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:34.356945 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.356905 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99"] Nov 04 00:56:34.357590 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.357559 6208 cpu_manager.go:414] "containerMap: removing container" podUID="f8c940b8-354d-44a5-a0fe-6738d544fb19" containerName="payload-processing" Nov 04 00:56:34.357590 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.357586 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c940b8-354d-44a5-a0fe-6738d544fb19" containerName="payload-processing" Nov 04 00:56:34.357806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.357624 6208 cpu_manager.go:414] "containerMap: removing container" podUID="83108053-a222-4664-a2d0-591c077a9ca3" containerName="maas-api" Nov 04 00:56:34.357806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.357633 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83108053-a222-4664-a2d0-591c077a9ca3" containerName="maas-api" Nov 04 00:56:34.357806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.357737 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c940b8-354d-44a5-a0fe-6738d544fb19" containerName="payload-processing" Nov 04 00:56:34.357806 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.357755 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="83108053-a222-4664-a2d0-591c077a9ca3" containerName="maas-api" Nov 04 00:56:34.362965 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.362931 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw"] Nov 04 00:56:34.363092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.363070 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:34.367209 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.367186 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:34.374619 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.374577 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99"] Nov 04 00:56:34.376089 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.376026 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b34223\"" Nov 04 00:56:34.376504 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.376484 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-15b71d4f-dockercfg-tcfps\"" Nov 04 00:56:34.379809 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.379786 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw"] Nov 04 00:56:34.386907 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.386880 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r"] Nov 04 00:56:34.392230 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.392210 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.399536 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.399463 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 00:56:34.405713 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.405679 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r"] Nov 04 00:56:34.437917 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.437881 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:34.472810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.472784 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:56:34.472941 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.472833 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:56:34.476352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.476333 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xflt\"" Nov 04 00:56:34.476352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.476333 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b34223-serving-cert\"" Nov 04 00:56:34.500837 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.500812 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvshd\" (UniqueName: \"kubernetes.io/projected/3effeace-7b08-4091-a01b-ff3850d3938c-kube-api-access-wvshd\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.500974 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.500848 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhn8l\" (UniqueName: \"kubernetes.io/projected/953e43c0-646f-446a-ab8a-a6b30a595b25-kube-api-access-lhn8l\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99\" (UID: \"953e43c0-646f-446a-ab8a-a6b30a595b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:34.500974 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.500871 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.501134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.500987 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166b8593-bdc1-402e-b11d-9a0e04e00ba4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw\" (UID: \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:34.501134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.501029 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vvz\" (UniqueName: \"kubernetes.io/projected/166b8593-bdc1-402e-b11d-9a0e04e00ba4-kube-api-access-j9vvz\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw\" (UID: \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:34.501134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.501100 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3effeace-7b08-4091-a01b-ff3850d3938c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.501134 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.501118 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.501366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.501151 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953e43c0-646f-446a-ab8a-a6b30a595b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99\" (UID: \"953e43c0-646f-446a-ab8a-a6b30a595b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:34.538179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.537965 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:34.602096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.602007 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953e43c0-646f-446a-ab8a-a6b30a595b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99\" (UID: \"953e43c0-646f-446a-ab8a-a6b30a595b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:34.602096 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.602087 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvshd\" (UniqueName: \"kubernetes.io/projected/3effeace-7b08-4091-a01b-ff3850d3938c-kube-api-access-wvshd\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.602388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.602122 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhn8l\" (UniqueName: \"kubernetes.io/projected/953e43c0-646f-446a-ab8a-a6b30a595b25-kube-api-access-lhn8l\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99\" (UID: \"953e43c0-646f-446a-ab8a-a6b30a595b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:34.602388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.602149 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.602388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.602192 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166b8593-bdc1-402e-b11d-9a0e04e00ba4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw\" (UID: \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:34.602388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.602259 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vvz\" (UniqueName: \"kubernetes.io/projected/166b8593-bdc1-402e-b11d-9a0e04e00ba4-kube-api-access-j9vvz\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw\" (UID: \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:34.602388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.602337 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3effeace-7b08-4091-a01b-ff3850d3938c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.602388 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.602364 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.603111 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.603025 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953e43c0-646f-446a-ab8a-a6b30a595b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99\" (UID: \"953e43c0-646f-446a-ab8a-a6b30a595b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:34.603235 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.603140 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3effeace-7b08-4091-a01b-ff3850d3938c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.603404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.603357 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166b8593-bdc1-402e-b11d-9a0e04e00ba4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw\" (UID: \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:34.605265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.605240 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.605402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.605306 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.614723 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.614701 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vvz\" (UniqueName: \"kubernetes.io/projected/166b8593-bdc1-402e-b11d-9a0e04e00ba4-kube-api-access-j9vvz\") pod \"payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw\" (UID: \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:34.614853 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.614838 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhn8l\" (UniqueName: \"kubernetes.io/projected/953e43c0-646f-446a-ab8a-a6b30a595b25-kube-api-access-lhn8l\") pod \"payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99\" (UID: \"953e43c0-646f-446a-ab8a-a6b30a595b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:34.614928 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.614905 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvshd\" (UniqueName: \"kubernetes.io/projected/3effeace-7b08-4091-a01b-ff3850d3938c-kube-api-access-wvshd\") pod \"maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.686746 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.686713 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:34.692152 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.692125 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:34.705587 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.705561 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:34.969518 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.969484 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99"] Nov 04 00:56:34.971454 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:56:34.971428 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953e43c0_646f_446a_ab8a_a6b30a595b25.slice/crio-d7c3bbc793502aea5994e7b9d44aefac670158d185420c532481392f672162d7 WatchSource:0}: Error finding container d7c3bbc793502aea5994e7b9d44aefac670158d185420c532481392f672162d7: Status 404 returned error can't find the container with id d7c3bbc793502aea5994e7b9d44aefac670158d185420c532481392f672162d7 Nov 04 00:56:34.985024 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:34.984754 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw"] Nov 04 00:56:34.994384 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:56:34.994361 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166b8593_bdc1_402e_b11d_9a0e04e00ba4.slice/crio-3d804c255f71e8d299d08d9f15f223fee1b0d9444c5d07451f0dc6a5ef2e5eaf WatchSource:0}: Error finding container 3d804c255f71e8d299d08d9f15f223fee1b0d9444c5d07451f0dc6a5ef2e5eaf: Status 404 returned error can't find the container with id 3d804c255f71e8d299d08d9f15f223fee1b0d9444c5d07451f0dc6a5ef2e5eaf Nov 04 00:56:35.023360 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.023333 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r"] Nov 04 00:56:35.024869 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:56:35.024845 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3effeace_7b08_4091_a01b_ff3850d3938c.slice/crio-0d2d01c1305d228f9b78ece0a6d1193f6674bf97714d1f16c0a964e11f08346b WatchSource:0}: Error finding container 0d2d01c1305d228f9b78ece0a6d1193f6674bf97714d1f16c0a964e11f08346b: Status 404 returned error can't find the container with id 0d2d01c1305d228f9b78ece0a6d1193f6674bf97714d1f16c0a964e11f08346b Nov 04 00:56:35.082890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.082856 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" event={"ID":"953e43c0-646f-446a-ab8a-a6b30a595b25","Type":"ContainerStarted","Data":"0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9"} Nov 04 00:56:35.083078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.082919 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" event={"ID":"953e43c0-646f-446a-ab8a-a6b30a595b25","Type":"ContainerStarted","Data":"d7c3bbc793502aea5994e7b9d44aefac670158d185420c532481392f672162d7"} Nov 04 00:56:35.083078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.082953 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:35.084564 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.084535 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" event={"ID":"166b8593-bdc1-402e-b11d-9a0e04e00ba4","Type":"ContainerStarted","Data":"986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb"} Nov 04 00:56:35.084695 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.084579 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" event={"ID":"166b8593-bdc1-402e-b11d-9a0e04e00ba4","Type":"ContainerStarted","Data":"3d804c255f71e8d299d08d9f15f223fee1b0d9444c5d07451f0dc6a5ef2e5eaf"} Nov 04 00:56:35.084695 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.084609 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:35.085780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.085757 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" event={"ID":"3effeace-7b08-4091-a01b-ff3850d3938c","Type":"ContainerStarted","Data":"0d2d01c1305d228f9b78ece0a6d1193f6674bf97714d1f16c0a964e11f08346b"} Nov 04 00:56:35.103242 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.103203 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" podStartSLOduration=1.103190863 podStartE2EDuration="1.103190863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:35.101827247 +0000 UTC m=+5213.924423408" watchObservedRunningTime="2026-11-04 00:56:35.103190863 +0000 UTC m=+5213.925787066" Nov 04 00:56:35.121016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.120978 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" podStartSLOduration=1.120967319 podStartE2EDuration="1.120967319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:35.118927216 +0000 UTC m=+5213.941523379" watchObservedRunningTime="2026-11-04 00:56:35.120967319 +0000 UTC m=+5213.943563481" Nov 04 00:56:35.259264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:35.259206 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:35.708287 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:56:35.708249 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:56:36.030063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:36.029825 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld"] Nov 04 00:56:36.032248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:36.032220 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-69gld"] Nov 04 00:56:36.036241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:36.036215 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:36.095659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:36.095622 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" event={"ID":"3effeace-7b08-4091-a01b-ff3850d3938c","Type":"ContainerStarted","Data":"9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62"} Nov 04 00:56:36.095860 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:36.095720 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:36.118130 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:36.118075 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" podStartSLOduration=1.623505481 podStartE2EDuration="2.118028953s" totalImagesPullingTime="494.523472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:36.116149812 +0000 UTC m=+5214.938745975" watchObservedRunningTime="2026-11-04 00:56:36.118028953 +0000 UTC m=+5214.940625115" Nov 04 00:56:36.932552 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:36.932507 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:37.294777 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:37.294745 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e991557-468c-4e3c-8068-29286832e39d" path="/var/lib/kubelet/pods/8e991557-468c-4e3c-8068-29286832e39d/volumes" Nov 04 00:56:37.333915 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:37.333880 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:37.930214 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:37.930170 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:38.695851 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:38.695820 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:38.897302 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:38.897251 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:39.645659 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:39.645609 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:40.426775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:40.426733 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:41.252595 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:41.252548 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:41.929154 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:41.929110 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:42.029684 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:42.029637 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:42.109091 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:42.109057 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:42.132327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:42.132263 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:42.821751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:42.821716 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:43.550184 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:43.550147 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:44.339560 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:44.339520 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:45.098782 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:45.098742 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:45.589135 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:45.589100 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:45.712348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:45.712309 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:45.928361 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:45.928265 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:46.098867 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:46.098837 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:46.099342 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:46.098914 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:46.662604 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:46.662569 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:47.334092 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:47.334059 6208 scope.go:122] "RemoveContainer" containerID="ad53373fec4405ffa5a5e779fa9e1260f22d7f8fcfd7fcaa79ea7e2059e552e7" Nov 04 00:56:47.343218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:47.343195 6208 scope.go:122] "RemoveContainer" containerID="970262f51e940375bfab4e09dd478427f2b59a7e74d5d40fd0ea2be499eab00d" Nov 04 00:56:47.351982 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:47.351964 6208 scope.go:122] "RemoveContainer" containerID="1243877ef76fab2976f9a33530805269f07d179abbf4a0a92d495ac1f3d2fdfe" Nov 04 00:56:47.360073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:47.360048 6208 scope.go:122] "RemoveContainer" containerID="4d3343c4f040e003c262047ee471936a32e21ed182149ea6df7da39f2e06abde" Nov 04 00:56:47.435407 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:47.435373 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:48.171292 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:48.171254 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:48.934391 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:48.934349 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:49.642048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:49.642005 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:50.424960 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:50.424901 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:51.026846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:51.026799 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk"] Nov 04 00:56:51.028793 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:51.028770 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wbssk"] Nov 04 00:56:51.142801 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:51.142754 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:51.294434 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:51.294355 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3" path="/var/lib/kubelet/pods/4b53f9e4-b9f1-40ce-8731-9ddfd9d241a3/volumes" Nov 04 00:56:51.921077 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:51.921013 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:52.647823 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:52.647785 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:53.418312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:53.418247 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:53.512104 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:53.512069 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw"] Nov 04 00:56:53.520592 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:53.520552 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99"] Nov 04 00:56:53.574048 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:53.573986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:53.674876 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:53.674790 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:54.142063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:54.142014 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:54.696866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:54.696830 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r"] Nov 04 00:56:54.697385 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:54.697098 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" podUID="3effeace-7b08-4091-a01b-ff3850d3938c" containerName="maas-api" containerID="cri-o://9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62" gracePeriod=30 Nov 04 00:56:54.744766 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:54.744733 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:54.961195 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:54.961164 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:54.992176 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:54.992144 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:55.012086 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.012053 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-maas-api-tls\") pod \"3effeace-7b08-4091-a01b-ff3850d3938c\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " Nov 04 00:56:55.012239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.012096 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-metrics-certs\") pod \"3effeace-7b08-4091-a01b-ff3850d3938c\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " Nov 04 00:56:55.012239 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.012214 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3effeace-7b08-4091-a01b-ff3850d3938c-kube-api-access-wvshd\" (UniqueName: \"kubernetes.io/projected/3effeace-7b08-4091-a01b-ff3850d3938c-kube-api-access-wvshd\") pod \"3effeace-7b08-4091-a01b-ff3850d3938c\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " Nov 04 00:56:55.012377 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.012242 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3effeace-7b08-4091-a01b-ff3850d3938c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3effeace-7b08-4091-a01b-ff3850d3938c-openshift-service-ca\") pod \"3effeace-7b08-4091-a01b-ff3850d3938c\" (UID: \"3effeace-7b08-4091-a01b-ff3850d3938c\") " Nov 04 00:56:55.012708 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.012670 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3effeace-7b08-4091-a01b-ff3850d3938c-openshift-service-ca" pod "3effeace-7b08-4091-a01b-ff3850d3938c" (UID: "3effeace-7b08-4091-a01b-ff3850d3938c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:56:55.014253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.014229 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-metrics-certs" pod "3effeace-7b08-4091-a01b-ff3850d3938c" (UID: "3effeace-7b08-4091-a01b-ff3850d3938c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:56:55.014366 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.014266 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-maas-api-tls" pod "3effeace-7b08-4091-a01b-ff3850d3938c" (UID: "3effeace-7b08-4091-a01b-ff3850d3938c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:56:55.014426 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.014405 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3effeace-7b08-4091-a01b-ff3850d3938c-kube-api-access-wvshd" pod "3effeace-7b08-4091-a01b-ff3850d3938c" (UID: "3effeace-7b08-4091-a01b-ff3850d3938c"). InnerVolumeSpecName "kube-api-access-wvshd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:56:55.103351 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.103311 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:55.113557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.113519 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:55.113557 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.113556 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3effeace-7b08-4091-a01b-ff3850d3938c-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:55.113751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.113570 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvshd\" (UniqueName: \"kubernetes.io/projected/3effeace-7b08-4091-a01b-ff3850d3938c-kube-api-access-wvshd\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:55.113751 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.113585 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3effeace-7b08-4091-a01b-ff3850d3938c-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:55.209605 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.209570 6208 generic.go:356] "Generic (PLEG): container finished" podID="3effeace-7b08-4091-a01b-ff3850d3938c" containerID="9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62" exitCode=0 Nov 04 00:56:55.209784 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.209620 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" Nov 04 00:56:55.209784 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.209670 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" event={"ID":"3effeace-7b08-4091-a01b-ff3850d3938c","Type":"ContainerDied","Data":"9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62"} Nov 04 00:56:55.209784 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.209727 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r" event={"ID":"3effeace-7b08-4091-a01b-ff3850d3938c","Type":"ContainerDied","Data":"0d2d01c1305d228f9b78ece0a6d1193f6674bf97714d1f16c0a964e11f08346b"} Nov 04 00:56:55.209784 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.209752 6208 scope.go:122] "RemoveContainer" containerID="9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62" Nov 04 00:56:55.220418 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.220397 6208 scope.go:122] "RemoveContainer" containerID="9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62" Nov 04 00:56:55.220694 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:56:55.220674 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62\": container with ID starting with 9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62 not found: ID does not exist" containerID="9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62" Nov 04 00:56:55.220748 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.220703 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62"} err="failed to get container status \"9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62\": rpc error: code = NotFound desc = could not find container \"9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62\": container with ID starting with 9ca76e947ec821e27a5fbaaed831990358d48c3b17a913ee93181fe3b2901a62 not found: ID does not exist" Nov 04 00:56:55.234734 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.234704 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r"] Nov 04 00:56:55.239081 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.239057 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b34223-99b7db447-nt59r"] Nov 04 00:56:55.293953 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.293919 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3effeace-7b08-4091-a01b-ff3850d3938c" path="/var/lib/kubelet/pods/3effeace-7b08-4091-a01b-ff3850d3938c/volumes" Nov 04 00:56:55.475750 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.475657 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl"] Nov 04 00:56:55.475931 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.475903 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" containerID="cri-o://1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad" gracePeriod=30 Nov 04 00:56:55.509872 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:55.509836 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:56.790246 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.790178 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4"] Nov 04 00:56:56.792515 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.792486 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgdt4"] Nov 04 00:56:56.817183 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.817145 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6"] Nov 04 00:56:56.817572 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.817557 6208 cpu_manager.go:414] "containerMap: removing container" podUID="3effeace-7b08-4091-a01b-ff3850d3938c" containerName="maas-api" Nov 04 00:56:56.817621 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.817572 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3effeace-7b08-4091-a01b-ff3850d3938c" containerName="maas-api" Nov 04 00:56:56.817671 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.817648 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="3effeace-7b08-4091-a01b-ff3850d3938c" containerName="maas-api" Nov 04 00:56:56.821984 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.821966 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:56.827961 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.827941 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:56:56.828168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.828147 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:56:56.828253 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.828147 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:56:56.829846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.829825 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:56:56.831516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.831341 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6"] Nov 04 00:56:56.930144 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.930109 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x8sh\" (UniqueName: \"kubernetes.io/projected/24bc803c-8cfa-4de7-95ae-7e33b11faa10-kube-api-access-8x8sh\") pod \"maas-api-revoke-sub-3d4156f1-rt2l6\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:56.930319 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.930154 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rt2l6\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:56.930394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:56.930317 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rt2l6\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:57.031016 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:57.030967 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rt2l6\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:57.031229 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:57.031074 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x8sh\" (UniqueName: \"kubernetes.io/projected/24bc803c-8cfa-4de7-95ae-7e33b11faa10-kube-api-access-8x8sh\") pod \"maas-api-revoke-sub-3d4156f1-rt2l6\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:57.031229 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:57.031119 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rt2l6\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:57.031973 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:57.031944 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rt2l6\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:57.033421 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:57.033392 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rt2l6\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:57.043609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:57.043541 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x8sh\" (UniqueName: \"kubernetes.io/projected/24bc803c-8cfa-4de7-95ae-7e33b11faa10-kube-api-access-8x8sh\") pod \"maas-api-revoke-sub-3d4156f1-rt2l6\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:57.136500 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:57.136460 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:57.299653 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:57.299571 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35713749-fbab-402c-b91b-431cade2c8eb" path="/var/lib/kubelet/pods/35713749-fbab-402c-b91b-431cade2c8eb/volumes" Nov 04 00:56:57.415576 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:57.415541 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6"] Nov 04 00:56:57.418772 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:56:57.418737 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bc803c_8cfa_4de7_95ae_7e33b11faa10.slice/crio-d9f1f936bb7e62ce2ec33c015bff62bd1f51ec6689c550db259a1f42f5f82467 WatchSource:0}: Error finding container d9f1f936bb7e62ce2ec33c015bff62bd1f51ec6689c550db259a1f42f5f82467: Status 404 returned error can't find the container with id d9f1f936bb7e62ce2ec33c015bff62bd1f51ec6689c550db259a1f42f5f82467 Nov 04 00:56:58.228830 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.228801 6208 generic.go:356] "Generic (PLEG): container finished" podID="24bc803c-8cfa-4de7-95ae-7e33b11faa10" containerID="49ee12f902a951c3d5f7a62f2a1883353a91d1811d108c831e056723c05065b6" exitCode=0 Nov 04 00:56:58.229256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.228883 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" event={"ID":"24bc803c-8cfa-4de7-95ae-7e33b11faa10","Type":"ContainerDied","Data":"49ee12f902a951c3d5f7a62f2a1883353a91d1811d108c831e056723c05065b6"} Nov 04 00:56:58.229256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.228936 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" event={"ID":"24bc803c-8cfa-4de7-95ae-7e33b11faa10","Type":"ContainerStarted","Data":"d9f1f936bb7e62ce2ec33c015bff62bd1f51ec6689c550db259a1f42f5f82467"} Nov 04 00:56:58.264379 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.264347 6208 patch_prober.go:29] interesting pod/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 00:56:58.264516 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.264398 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:56:58.512846 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.512792 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" podUID="166b8593-bdc1-402e-b11d-9a0e04e00ba4" containerName="payload-processing" containerID="cri-o://986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb" gracePeriod=25 Nov 04 00:56:58.521338 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.521281 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" podUID="953e43c0-646f-446a-ab8a-a6b30a595b25" containerName="payload-pre-processing" containerID="cri-o://0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9" gracePeriod=25 Nov 04 00:56:58.523327 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.523286 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:58.626688 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.626646 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:56:58.769381 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.769322 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:58.802451 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.802422 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:58.849315 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.849281 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166b8593-bdc1-402e-b11d-9a0e04e00ba4-kube-api-access-j9vvz\" (UniqueName: \"kubernetes.io/projected/166b8593-bdc1-402e-b11d-9a0e04e00ba4-kube-api-access-j9vvz\") pod \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\" (UID: \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\") " Nov 04 00:56:58.849497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.849344 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953e43c0-646f-446a-ab8a-a6b30a595b25-kube-api-access-lhn8l\" (UniqueName: \"kubernetes.io/projected/953e43c0-646f-446a-ab8a-a6b30a595b25-kube-api-access-lhn8l\") pod \"953e43c0-646f-446a-ab8a-a6b30a595b25\" (UID: \"953e43c0-646f-446a-ab8a-a6b30a595b25\") " Nov 04 00:56:58.849497 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.849448 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953e43c0-646f-446a-ab8a-a6b30a595b25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953e43c0-646f-446a-ab8a-a6b30a595b25-plugins-config-volume\") pod \"953e43c0-646f-446a-ab8a-a6b30a595b25\" (UID: \"953e43c0-646f-446a-ab8a-a6b30a595b25\") " Nov 04 00:56:58.849637 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.849515 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166b8593-bdc1-402e-b11d-9a0e04e00ba4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166b8593-bdc1-402e-b11d-9a0e04e00ba4-plugins-config-volume\") pod \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\" (UID: \"166b8593-bdc1-402e-b11d-9a0e04e00ba4\") " Nov 04 00:56:58.849929 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.849891 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953e43c0-646f-446a-ab8a-a6b30a595b25-plugins-config-volume" pod "953e43c0-646f-446a-ab8a-a6b30a595b25" (UID: "953e43c0-646f-446a-ab8a-a6b30a595b25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:56:58.850080 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.849954 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166b8593-bdc1-402e-b11d-9a0e04e00ba4-plugins-config-volume" pod "166b8593-bdc1-402e-b11d-9a0e04e00ba4" (UID: "166b8593-bdc1-402e-b11d-9a0e04e00ba4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:56:58.851526 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.851501 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166b8593-bdc1-402e-b11d-9a0e04e00ba4-kube-api-access-j9vvz" pod "166b8593-bdc1-402e-b11d-9a0e04e00ba4" (UID: "166b8593-bdc1-402e-b11d-9a0e04e00ba4"). InnerVolumeSpecName "kube-api-access-j9vvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:56:58.851792 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.851772 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953e43c0-646f-446a-ab8a-a6b30a595b25-kube-api-access-lhn8l" pod "953e43c0-646f-446a-ab8a-a6b30a595b25" (UID: "953e43c0-646f-446a-ab8a-a6b30a595b25"). InnerVolumeSpecName "kube-api-access-lhn8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:56:58.951275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.951238 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953e43c0-646f-446a-ab8a-a6b30a595b25-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:58.951275 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.951272 6208 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166b8593-bdc1-402e-b11d-9a0e04e00ba4-plugins-config-volume\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:58.951476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.951286 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9vvz\" (UniqueName: \"kubernetes.io/projected/166b8593-bdc1-402e-b11d-9a0e04e00ba4-kube-api-access-j9vvz\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:58.951476 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:58.951301 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhn8l\" (UniqueName: \"kubernetes.io/projected/953e43c0-646f-446a-ab8a-a6b30a595b25-kube-api-access-lhn8l\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:59.237108 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.237072 6208 generic.go:356] "Generic (PLEG): container finished" podID="166b8593-bdc1-402e-b11d-9a0e04e00ba4" containerID="986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb" exitCode=0 Nov 04 00:56:59.237567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.237119 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" Nov 04 00:56:59.237567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.237158 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" event={"ID":"166b8593-bdc1-402e-b11d-9a0e04e00ba4","Type":"ContainerDied","Data":"986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb"} Nov 04 00:56:59.237567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.237211 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw" event={"ID":"166b8593-bdc1-402e-b11d-9a0e04e00ba4","Type":"ContainerDied","Data":"3d804c255f71e8d299d08d9f15f223fee1b0d9444c5d07451f0dc6a5ef2e5eaf"} Nov 04 00:56:59.237567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.237238 6208 scope.go:122] "RemoveContainer" containerID="986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb" Nov 04 00:56:59.238584 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.238560 6208 generic.go:356] "Generic (PLEG): container finished" podID="953e43c0-646f-446a-ab8a-a6b30a595b25" containerID="0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9" exitCode=0 Nov 04 00:56:59.238704 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.238607 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" Nov 04 00:56:59.238704 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.238611 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" event={"ID":"953e43c0-646f-446a-ab8a-a6b30a595b25","Type":"ContainerDied","Data":"0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9"} Nov 04 00:56:59.238704 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.238661 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99" event={"ID":"953e43c0-646f-446a-ab8a-a6b30a595b25","Type":"ContainerDied","Data":"d7c3bbc793502aea5994e7b9d44aefac670158d185420c532481392f672162d7"} Nov 04 00:56:59.249244 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.249221 6208 scope.go:122] "RemoveContainer" containerID="986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb" Nov 04 00:56:59.249537 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:56:59.249506 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb\": container with ID starting with 986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb not found: ID does not exist" containerID="986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb" Nov 04 00:56:59.249609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.249550 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb"} err="failed to get container status \"986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb\": rpc error: code = NotFound desc = could not find container \"986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb\": container with ID starting with 986f1e6672810a3d2b75f299db437bfdfaef6d4dd51658afc3bda84432ce4ceb not found: ID does not exist" Nov 04 00:56:59.249609 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.249574 6208 scope.go:122] "RemoveContainer" containerID="0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9" Nov 04 00:56:59.258490 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.258472 6208 scope.go:122] "RemoveContainer" containerID="0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9" Nov 04 00:56:59.258848 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:56:59.258829 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9\": container with ID starting with 0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9 not found: ID does not exist" containerID="0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9" Nov 04 00:56:59.258911 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.258862 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9"} err="failed to get container status \"0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9\": rpc error: code = NotFound desc = could not find container \"0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9\": container with ID starting with 0bff238d1777d2e81a316a5c508bcbbdf8aa352a3b82c79c7255ff14c9a63bc9 not found: ID does not exist" Nov 04 00:56:59.267827 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.267791 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw"] Nov 04 00:56:59.270073 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.270048 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b34223-6cff48b9dmbdsw"] Nov 04 00:56:59.284570 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.284539 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99"] Nov 04 00:56:59.295521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.295458 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166b8593-bdc1-402e-b11d-9a0e04e00ba4" path="/var/lib/kubelet/pods/166b8593-bdc1-402e-b11d-9a0e04e00ba4/volumes" Nov 04 00:56:59.295754 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.295740 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b34223-6fdcfl8s99"] Nov 04 00:56:59.380768 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.380742 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:56:59.455768 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.455737 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-xks-ca\") pod \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " Nov 04 00:56:59.455936 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.455918 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24bc803c-8cfa-4de7-95ae-7e33b11faa10-kube-api-access-8x8sh\" (UniqueName: \"kubernetes.io/projected/24bc803c-8cfa-4de7-95ae-7e33b11faa10-kube-api-access-8x8sh\") pod \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " Nov 04 00:56:59.455986 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.455963 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-openshift-ca\") pod \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\" (UID: \"24bc803c-8cfa-4de7-95ae-7e33b11faa10\") " Nov 04 00:56:59.456340 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.456316 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-openshift-ca" pod "24bc803c-8cfa-4de7-95ae-7e33b11faa10" (UID: "24bc803c-8cfa-4de7-95ae-7e33b11faa10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:56:59.457834 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.457805 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-xks-ca" pod "24bc803c-8cfa-4de7-95ae-7e33b11faa10" (UID: "24bc803c-8cfa-4de7-95ae-7e33b11faa10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:56:59.457939 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.457882 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bc803c-8cfa-4de7-95ae-7e33b11faa10-kube-api-access-8x8sh" pod "24bc803c-8cfa-4de7-95ae-7e33b11faa10" (UID: "24bc803c-8cfa-4de7-95ae-7e33b11faa10"). InnerVolumeSpecName "kube-api-access-8x8sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:56:59.557093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.556995 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x8sh\" (UniqueName: \"kubernetes.io/projected/24bc803c-8cfa-4de7-95ae-7e33b11faa10-kube-api-access-8x8sh\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:59.557093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.557024 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:59.557093 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.557064 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24bc803c-8cfa-4de7-95ae-7e33b11faa10-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:56:59.759664 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:56:59.759626 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:00.247348 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.247315 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" event={"ID":"24bc803c-8cfa-4de7-95ae-7e33b11faa10","Type":"ContainerDied","Data":"d9f1f936bb7e62ce2ec33c015bff62bd1f51ec6689c550db259a1f42f5f82467"} Nov 04 00:57:00.247767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.247350 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6" Nov 04 00:57:00.247767 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.247366 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9f1f936bb7e62ce2ec33c015bff62bd1f51ec6689c550db259a1f42f5f82467" Nov 04 00:57:00.729011 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.728985 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:57:00.768311 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768224 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-envoy\") pod \"50b989dd-410a-4fb1-91b8-8af60c1186bd\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " Nov 04 00:57:00.768488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768312 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-podinfo\") pod \"50b989dd-410a-4fb1-91b8-8af60c1186bd\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " Nov 04 00:57:00.768488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768353 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-token\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-token\") pod \"50b989dd-410a-4fb1-91b8-8af60c1186bd\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " Nov 04 00:57:00.768488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768398 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-data\") pod \"50b989dd-410a-4fb1-91b8-8af60c1186bd\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " Nov 04 00:57:00.768488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768433 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b989dd-410a-4fb1-91b8-8af60c1186bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b989dd-410a-4fb1-91b8-8af60c1186bd-istiod-ca-cert\") pod \"50b989dd-410a-4fb1-91b8-8af60c1186bd\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " Nov 04 00:57:00.768488 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768476 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-kube-api-access-wrws7\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-kube-api-access-wrws7\") pod \"50b989dd-410a-4fb1-91b8-8af60c1186bd\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " Nov 04 00:57:00.768775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768518 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-socket\") pod \"50b989dd-410a-4fb1-91b8-8af60c1186bd\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " Nov 04 00:57:00.768775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768554 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-certs\") pod \"50b989dd-410a-4fb1-91b8-8af60c1186bd\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " Nov 04 00:57:00.768775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768613 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-credential-socket\") pod \"50b989dd-410a-4fb1-91b8-8af60c1186bd\" (UID: \"50b989dd-410a-4fb1-91b8-8af60c1186bd\") " Nov 04 00:57:00.768775 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768671 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-data" pod "50b989dd-410a-4fb1-91b8-8af60c1186bd" (UID: "50b989dd-410a-4fb1-91b8-8af60c1186bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:57:00.768975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768882 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b989dd-410a-4fb1-91b8-8af60c1186bd-istiod-ca-cert" pod "50b989dd-410a-4fb1-91b8-8af60c1186bd" (UID: "50b989dd-410a-4fb1-91b8-8af60c1186bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:57:00.768975 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.768940 6208 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-data\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:00.769139 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.769002 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-certs" pod "50b989dd-410a-4fb1-91b8-8af60c1186bd" (UID: "50b989dd-410a-4fb1-91b8-8af60c1186bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:57:00.769248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.769208 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-credential-socket" pod "50b989dd-410a-4fb1-91b8-8af60c1186bd" (UID: "50b989dd-410a-4fb1-91b8-8af60c1186bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:57:00.769438 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.769414 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-socket" pod "50b989dd-410a-4fb1-91b8-8af60c1186bd" (UID: "50b989dd-410a-4fb1-91b8-8af60c1186bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:57:00.770597 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.770576 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-envoy" pod "50b989dd-410a-4fb1-91b8-8af60c1186bd" (UID: "50b989dd-410a-4fb1-91b8-8af60c1186bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:57:00.770902 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.770877 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-podinfo" pod "50b989dd-410a-4fb1-91b8-8af60c1186bd" (UID: "50b989dd-410a-4fb1-91b8-8af60c1186bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 00:57:00.771179 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.771158 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-kube-api-access-wrws7" pod "50b989dd-410a-4fb1-91b8-8af60c1186bd" (UID: "50b989dd-410a-4fb1-91b8-8af60c1186bd"). InnerVolumeSpecName "kube-api-access-wrws7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:57:00.771285 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.771186 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-token" pod "50b989dd-410a-4fb1-91b8-8af60c1186bd" (UID: "50b989dd-410a-4fb1-91b8-8af60c1186bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:57:00.869839 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.869803 6208 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-credential-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:00.869839 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.869836 6208 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-envoy\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:00.869839 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.869847 6208 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-podinfo\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:00.870141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.869858 6208 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-istio-token\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:00.870141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.869869 6208 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b989dd-410a-4fb1-91b8-8af60c1186bd-istiod-ca-cert\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:00.870141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.869879 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrws7\" (UniqueName: \"kubernetes.io/projected/50b989dd-410a-4fb1-91b8-8af60c1186bd-kube-api-access-wrws7\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:00.870141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.869889 6208 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-socket\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:00.870141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:00.869898 6208 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b989dd-410a-4fb1-91b8-8af60c1186bd-workload-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:01.254779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.254742 6208 generic.go:356] "Generic (PLEG): container finished" podID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerID="1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad" exitCode=0 Nov 04 00:57:01.255241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.254789 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" Nov 04 00:57:01.255241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.254796 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" event={"ID":"50b989dd-410a-4fb1-91b8-8af60c1186bd","Type":"ContainerDied","Data":"1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad"} Nov 04 00:57:01.255241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.254922 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl" event={"ID":"50b989dd-410a-4fb1-91b8-8af60c1186bd","Type":"ContainerDied","Data":"b369f00f7257f39906bd2f735df5fb6850baa69588da5cee91c27536ae0b471b"} Nov 04 00:57:01.255241 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.254955 6208 scope.go:122] "RemoveContainer" containerID="1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad" Nov 04 00:57:01.264926 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.264907 6208 scope.go:122] "RemoveContainer" containerID="30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf" Nov 04 00:57:01.273279 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.273257 6208 scope.go:122] "RemoveContainer" containerID="1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad" Nov 04 00:57:01.273585 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:57:01.273553 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad\": container with ID starting with 1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad not found: ID does not exist" containerID="1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad" Nov 04 00:57:01.273687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.273584 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad"} err="failed to get container status \"1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad\": rpc error: code = NotFound desc = could not find container \"1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad\": container with ID starting with 1311a04f54bfc18dc0c6fe9e7330d97d7c78175c07b4a19d0d1353d1fd1bf4ad not found: ID does not exist" Nov 04 00:57:01.273687 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.273602 6208 scope.go:122] "RemoveContainer" containerID="30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf" Nov 04 00:57:01.273853 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:57:01.273833 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf\": container with ID starting with 30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf not found: ID does not exist" containerID="30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf" Nov 04 00:57:01.273906 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.273859 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf"} err="failed to get container status \"30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf\": rpc error: code = NotFound desc = could not find container \"30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf\": container with ID starting with 30e659c743328ad8e6a76bbf4cc3b620e7b3e317bb5cc709dee77efab2b174bf not found: ID does not exist" Nov 04 00:57:01.279797 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.279772 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl"] Nov 04 00:57:01.283628 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.283601 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b34223-openshift-default-796bf689852lxxl"] Nov 04 00:57:01.292780 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.292754 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" path="/var/lib/kubelet/pods/50b989dd-410a-4fb1-91b8-8af60c1186bd/volumes" Nov 04 00:57:01.293218 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:01.293199 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953e43c0-646f-446a-ab8a-a6b30a595b25" path="/var/lib/kubelet/pods/953e43c0-646f-446a-ab8a-a6b30a595b25/volumes" Nov 04 00:57:05.703775 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:57:05.703734 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:57:06.028728 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:06.028693 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd"] Nov 04 00:57:06.031233 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:06.031196 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tqhgd"] Nov 04 00:57:07.292260 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:07.292227 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ceb16cd-fbaa-4904-a7af-593e33494564" path="/var/lib/kubelet/pods/9ceb16cd-fbaa-4904-a7af-593e33494564/volumes" Nov 04 00:57:09.372694 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:09.372652 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:20.732524 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:20.732490 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:25.130921 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:25.130875 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:35.707219 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:57:35.707176 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:57:41.358920 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:41.358883 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:41.459223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:41.459188 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:47.407200 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:47.407165 6208 scope.go:122] "RemoveContainer" containerID="b95b8ef465932c2f9ea7a1ee1ca3e11426963ea69fb7a2da6de7460a01a205e5" Nov 04 00:57:47.416447 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:47.416419 6208 scope.go:122] "RemoveContainer" containerID="9a848c9074104ba03b38ff85a621b93e91758bb08fba5ff2e0b4bbadb7b51eb2" Nov 04 00:57:47.425274 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:57:47.425252 6208 scope.go:122] "RemoveContainer" containerID="f47d5ba904a5e66260cc75181902c0eb2e1a86af97298b0299a52950264c6500" Nov 04 00:58:05.704683 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:58:05.704648 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:58:09.415606 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:09.415561 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:15.028657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:15.028620 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks"] Nov 04 00:58:15.031430 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:15.031400 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-gccks"] Nov 04 00:58:15.293357 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:15.293285 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a9a396-9047-4f41-8d49-f739f0886037" path="/var/lib/kubelet/pods/d5a9a396-9047-4f41-8d49-f739f0886037/volumes" Nov 04 00:58:20.790528 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:20.790438 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:24.026356 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:24.026318 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z"] Nov 04 00:58:24.028675 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:24.028649 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-hv27z"] Nov 04 00:58:25.187883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:25.187849 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:25.292615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:25.292579 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ec63b4-767f-415d-8bc6-ac95870d8c9c" path="/var/lib/kubelet/pods/74ec63b4-767f-415d-8bc6-ac95870d8c9c/volumes" Nov 04 00:58:35.704237 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:58:35.704196 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:58:44.629051 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.628988 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn"] Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629384 6208 cpu_manager.go:414] "containerMap: removing container" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629397 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629411 6208 cpu_manager.go:414] "containerMap: removing container" podUID="166b8593-bdc1-402e-b11d-9a0e04e00ba4" containerName="payload-processing" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629417 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166b8593-bdc1-402e-b11d-9a0e04e00ba4" containerName="payload-processing" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629436 6208 cpu_manager.go:414] "containerMap: removing container" podUID="953e43c0-646f-446a-ab8a-a6b30a595b25" containerName="payload-pre-processing" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629441 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953e43c0-646f-446a-ab8a-a6b30a595b25" containerName="payload-pre-processing" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629459 6208 cpu_manager.go:414] "containerMap: removing container" podUID="24bc803c-8cfa-4de7-95ae-7e33b11faa10" containerName="revoke-keys" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629465 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24bc803c-8cfa-4de7-95ae-7e33b11faa10" containerName="revoke-keys" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629518 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="953e43c0-646f-446a-ab8a-a6b30a595b25" containerName="payload-pre-processing" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629532 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="24bc803c-8cfa-4de7-95ae-7e33b11faa10" containerName="revoke-keys" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629546 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629557 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" Nov 04 00:58:44.629567 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.629566 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="166b8593-bdc1-402e-b11d-9a0e04e00ba4" containerName="payload-processing" Nov 04 00:58:44.632736 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.632718 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.640168 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.640135 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:58:44.640697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.640671 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:58:44.640697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.640672 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:58:44.640840 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.640765 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:58:44.643072 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.643019 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn"] Nov 04 00:58:44.720624 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.720590 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cnwfn\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.720838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.720644 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cnwfn\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.720838 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.720761 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2f7q\" (UniqueName: \"kubernetes.io/projected/67a88294-9db4-4ce5-bb00-91f9d8cba085-kube-api-access-b2f7q\") pod \"maas-api-revoke-sub-d576e68b-cnwfn\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.822188 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.822145 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cnwfn\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.822406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.822216 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cnwfn\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.822406 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.822276 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2f7q\" (UniqueName: \"kubernetes.io/projected/67a88294-9db4-4ce5-bb00-91f9d8cba085-kube-api-access-b2f7q\") pod \"maas-api-revoke-sub-d576e68b-cnwfn\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.822919 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.822892 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cnwfn\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.824658 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.824633 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cnwfn\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.834977 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.834952 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2f7q\" (UniqueName: \"kubernetes.io/projected/67a88294-9db4-4ce5-bb00-91f9d8cba085-kube-api-access-b2f7q\") pod \"maas-api-revoke-sub-d576e68b-cnwfn\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:44.947615 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:44.947513 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:45.224540 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:45.224512 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn"] Nov 04 00:58:45.226822 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:58:45.226792 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a88294_9db4_4ce5_bb00_91f9d8cba085.slice/crio-ba5295633b99c568dac22ee8740558be0932f2181923c4d33089dbf29ee02f4d WatchSource:0}: Error finding container ba5295633b99c568dac22ee8740558be0932f2181923c4d33089dbf29ee02f4d: Status 404 returned error can't find the container with id ba5295633b99c568dac22ee8740558be0932f2181923c4d33089dbf29ee02f4d Nov 04 00:58:45.896620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:45.896529 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" event={"ID":"67a88294-9db4-4ce5-bb00-91f9d8cba085","Type":"ContainerStarted","Data":"adca671b5422dd8559aa5df8784f51d62528eea525b0e3ce55a24356e5c2562d"} Nov 04 00:58:45.896620 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:45.896588 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" event={"ID":"67a88294-9db4-4ce5-bb00-91f9d8cba085","Type":"ContainerStarted","Data":"ba5295633b99c568dac22ee8740558be0932f2181923c4d33089dbf29ee02f4d"} Nov 04 00:58:45.918365 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:45.918314 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" podStartSLOduration=1.918300586 podStartE2EDuration="1.918300586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:45.916445424 +0000 UTC m=+5344.739041586" watchObservedRunningTime="2026-11-04 00:58:45.918300586 +0000 UTC m=+5344.740896745" Nov 04 00:58:46.902936 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:46.902896 6208 generic.go:356] "Generic (PLEG): container finished" podID="67a88294-9db4-4ce5-bb00-91f9d8cba085" containerID="adca671b5422dd8559aa5df8784f51d62528eea525b0e3ce55a24356e5c2562d" exitCode=0 Nov 04 00:58:46.903353 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:46.902987 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" event={"ID":"67a88294-9db4-4ce5-bb00-91f9d8cba085","Type":"ContainerDied","Data":"adca671b5422dd8559aa5df8784f51d62528eea525b0e3ce55a24356e5c2562d"} Nov 04 00:58:47.477085 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:47.477050 6208 scope.go:122] "RemoveContainer" containerID="35354d453ad07d28cbe9fa8c38af34784ca8a427f15c75ae375886ae15168154" Nov 04 00:58:47.486884 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:47.486857 6208 scope.go:122] "RemoveContainer" containerID="fd2097b486f182dd16a6128afe69811a56d6a6b0bdd7b3de10c2978e729edf68" Nov 04 00:58:47.494885 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:47.494865 6208 scope.go:122] "RemoveContainer" containerID="00b29a322c094e8232887e5e4ba5b22e702f2196107161bee1c11142cd7ad6b3" Nov 04 00:58:47.502779 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:47.502757 6208 scope.go:122] "RemoveContainer" containerID="ddb7a757bb8db4f89845b79139af2552b6fbcfbf20e7159114aa3efe9c02677b" Nov 04 00:58:48.044740 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.044713 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:48.151703 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.151667 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-xks-ca\") pod \"67a88294-9db4-4ce5-bb00-91f9d8cba085\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " Nov 04 00:58:48.151866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.151785 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-openshift-ca\") pod \"67a88294-9db4-4ce5-bb00-91f9d8cba085\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " Nov 04 00:58:48.151866 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.151829 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a88294-9db4-4ce5-bb00-91f9d8cba085-kube-api-access-b2f7q\" (UniqueName: \"kubernetes.io/projected/67a88294-9db4-4ce5-bb00-91f9d8cba085-kube-api-access-b2f7q\") pod \"67a88294-9db4-4ce5-bb00-91f9d8cba085\" (UID: \"67a88294-9db4-4ce5-bb00-91f9d8cba085\") " Nov 04 00:58:48.152177 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.152153 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-openshift-ca" pod "67a88294-9db4-4ce5-bb00-91f9d8cba085" (UID: "67a88294-9db4-4ce5-bb00-91f9d8cba085"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:48.153899 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.153871 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-xks-ca" pod "67a88294-9db4-4ce5-bb00-91f9d8cba085" (UID: "67a88294-9db4-4ce5-bb00-91f9d8cba085"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:48.154015 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.153937 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a88294-9db4-4ce5-bb00-91f9d8cba085-kube-api-access-b2f7q" pod "67a88294-9db4-4ce5-bb00-91f9d8cba085" (UID: "67a88294-9db4-4ce5-bb00-91f9d8cba085"). InnerVolumeSpecName "kube-api-access-b2f7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:48.253422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.253386 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:48.253422 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.253416 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2f7q\" (UniqueName: \"kubernetes.io/projected/67a88294-9db4-4ce5-bb00-91f9d8cba085-kube-api-access-b2f7q\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:48.253610 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.253430 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a88294-9db4-4ce5-bb00-91f9d8cba085-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:48.914010 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.913975 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" Nov 04 00:58:48.914223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.914008 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn" event={"ID":"67a88294-9db4-4ce5-bb00-91f9d8cba085","Type":"ContainerDied","Data":"ba5295633b99c568dac22ee8740558be0932f2181923c4d33089dbf29ee02f4d"} Nov 04 00:58:48.914223 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:58:48.914102 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba5295633b99c568dac22ee8740558be0932f2181923c4d33089dbf29ee02f4d" Nov 04 00:59:01.173870 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.173833 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd"] Nov 04 00:59:01.174312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.174273 6208 cpu_manager.go:414] "containerMap: removing container" podUID="67a88294-9db4-4ce5-bb00-91f9d8cba085" containerName="revoke-keys" Nov 04 00:59:01.174312 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.174286 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a88294-9db4-4ce5-bb00-91f9d8cba085" containerName="revoke-keys" Nov 04 00:59:01.174390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.174316 6208 cpu_manager.go:414] "containerMap: removing container" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" Nov 04 00:59:01.174390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.174322 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b989dd-410a-4fb1-91b8-8af60c1186bd" containerName="istio-proxy" Nov 04 00:59:01.174390 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.174384 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a88294-9db4-4ce5-bb00-91f9d8cba085" containerName="revoke-keys" Nov 04 00:59:01.177425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.177402 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.184350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.184111 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 00:59:01.184350 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.184179 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:59:01.186072 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.185733 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:59:01.189005 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.188980 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd"] Nov 04 00:59:01.189172 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.189150 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 00:59:01.328771 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.328735 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l8hd\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.328954 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.328868 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf96f\" (UniqueName: \"kubernetes.io/projected/2300905c-5cae-4639-8f34-469c832a18f1-kube-api-access-jf96f\") pod \"maas-api-revoke-sub-d2bc93bc-2l8hd\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.329007 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.328960 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l8hd\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.429696 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.429634 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf96f\" (UniqueName: \"kubernetes.io/projected/2300905c-5cae-4639-8f34-469c832a18f1-kube-api-access-jf96f\") pod \"maas-api-revoke-sub-d2bc93bc-2l8hd\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.429817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.429706 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l8hd\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.429817 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.429730 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l8hd\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.430521 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.430494 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l8hd\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.432063 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.432028 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l8hd\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.442219 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.442198 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf96f\" (UniqueName: \"kubernetes.io/projected/2300905c-5cae-4639-8f34-469c832a18f1-kube-api-access-jf96f\") pod \"maas-api-revoke-sub-d2bc93bc-2l8hd\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.493075 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.493049 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:01.770821 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:01.770794 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd"] Nov 04 00:59:01.773027 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:59:01.773001 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2300905c_5cae_4639_8f34_469c832a18f1.slice/crio-9908ab7982d238e6001fc40b9d8ea8db51d95372ddc7eb7830fea0e306c32af0 WatchSource:0}: Error finding container 9908ab7982d238e6001fc40b9d8ea8db51d95372ddc7eb7830fea0e306c32af0: Status 404 returned error can't find the container with id 9908ab7982d238e6001fc40b9d8ea8db51d95372ddc7eb7830fea0e306c32af0 Nov 04 00:59:02.022170 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:02.022073 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" event={"ID":"2300905c-5cae-4639-8f34-469c832a18f1","Type":"ContainerStarted","Data":"9908ab7982d238e6001fc40b9d8ea8db51d95372ddc7eb7830fea0e306c32af0"} Nov 04 00:59:03.028473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:03.028432 6208 generic.go:356] "Generic (PLEG): container finished" podID="2300905c-5cae-4639-8f34-469c832a18f1" containerID="6138cd8aef94b04dcb4975ca966ed061fc54d9e65a0ca111f21f9dc133be0c24" exitCode=0 Nov 04 00:59:03.028473 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:03.028478 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" event={"ID":"2300905c-5cae-4639-8f34-469c832a18f1","Type":"ContainerDied","Data":"6138cd8aef94b04dcb4975ca966ed061fc54d9e65a0ca111f21f9dc133be0c24"} Nov 04 00:59:04.167084 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.167049 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:04.359898 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.359803 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-xks-ca\") pod \"2300905c-5cae-4639-8f34-469c832a18f1\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " Nov 04 00:59:04.360205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.359924 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2300905c-5cae-4639-8f34-469c832a18f1-kube-api-access-jf96f\" (UniqueName: \"kubernetes.io/projected/2300905c-5cae-4639-8f34-469c832a18f1-kube-api-access-jf96f\") pod \"2300905c-5cae-4639-8f34-469c832a18f1\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " Nov 04 00:59:04.360205 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.359967 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-openshift-ca\") pod \"2300905c-5cae-4639-8f34-469c832a18f1\" (UID: \"2300905c-5cae-4639-8f34-469c832a18f1\") " Nov 04 00:59:04.360396 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.360364 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-openshift-ca" pod "2300905c-5cae-4639-8f34-469c832a18f1" (UID: "2300905c-5cae-4639-8f34-469c832a18f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:04.362138 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.362110 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-xks-ca" pod "2300905c-5cae-4639-8f34-469c832a18f1" (UID: "2300905c-5cae-4639-8f34-469c832a18f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:59:04.362265 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.362142 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2300905c-5cae-4639-8f34-469c832a18f1-kube-api-access-jf96f" pod "2300905c-5cae-4639-8f34-469c832a18f1" (UID: "2300905c-5cae-4639-8f34-469c832a18f1"). InnerVolumeSpecName "kube-api-access-jf96f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:59:04.461346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.461307 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:04.461346 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.461345 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf96f\" (UniqueName: \"kubernetes.io/projected/2300905c-5cae-4639-8f34-469c832a18f1-kube-api-access-jf96f\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:04.461551 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:04.461361 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2300905c-5cae-4639-8f34-469c832a18f1-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:05.040574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:05.040541 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" Nov 04 00:59:05.040574 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:05.040565 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd" event={"ID":"2300905c-5cae-4639-8f34-469c832a18f1","Type":"ContainerDied","Data":"9908ab7982d238e6001fc40b9d8ea8db51d95372ddc7eb7830fea0e306c32af0"} Nov 04 00:59:05.040796 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:05.040608 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9908ab7982d238e6001fc40b9d8ea8db51d95372ddc7eb7830fea0e306c32af0" Nov 04 00:59:05.705023 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:59:05.704981 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:59:09.470818 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:09.470781 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:13.375960 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.375920 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-gnhv2"] Nov 04 00:59:13.376394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.376351 6208 cpu_manager.go:414] "containerMap: removing container" podUID="2300905c-5cae-4639-8f34-469c832a18f1" containerName="revoke-keys" Nov 04 00:59:13.376394 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.376373 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2300905c-5cae-4639-8f34-469c832a18f1" containerName="revoke-keys" Nov 04 00:59:13.376481 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.376455 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="2300905c-5cae-4639-8f34-469c832a18f1" containerName="revoke-keys" Nov 04 00:59:13.379725 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.379704 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.385710 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.385689 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 00:59:13.385710 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.385691 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 00:59:13.385903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.385759 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xflt\"" Nov 04 00:59:13.385903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.385825 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 00:59:13.385903 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.385870 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 00:59:13.389651 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.389629 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-gnhv2"] Nov 04 00:59:13.395156 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.395134 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl"] Nov 04 00:59:13.398718 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.398695 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.409322 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.409297 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl"] Nov 04 00:59:13.440959 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.440927 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25346f5b-b129-4e2a-ac50-4b8d7858c91e-metrics-certs\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.441123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.440967 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-metrics-certs\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.441123 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.441095 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25346f5b-b129-4e2a-ac50-4b8d7858c91e-openshift-service-ca\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.441244 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.441128 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25346f5b-b129-4e2a-ac50-4b8d7858c91e-maas-api-tls\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.441244 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.441155 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crh7z\" (UniqueName: \"kubernetes.io/projected/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-kube-api-access-crh7z\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.441352 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.441258 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-maas-api-tls\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.441402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.441354 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-openshift-service-ca\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.441402 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.441395 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmlv\" (UniqueName: \"kubernetes.io/projected/25346f5b-b129-4e2a-ac50-4b8d7858c91e-kube-api-access-hhmlv\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.542883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.542847 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25346f5b-b129-4e2a-ac50-4b8d7858c91e-openshift-service-ca\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.542883 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.542896 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25346f5b-b129-4e2a-ac50-4b8d7858c91e-maas-api-tls\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.543194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.543014 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crh7z\" (UniqueName: \"kubernetes.io/projected/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-kube-api-access-crh7z\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.543194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.543092 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-maas-api-tls\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.543194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.543147 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-openshift-service-ca\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.543194 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.543188 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmlv\" (UniqueName: \"kubernetes.io/projected/25346f5b-b129-4e2a-ac50-4b8d7858c91e-kube-api-access-hhmlv\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.543425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.543219 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25346f5b-b129-4e2a-ac50-4b8d7858c91e-metrics-certs\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.543425 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.543246 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-metrics-certs\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.543741 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.543715 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25346f5b-b129-4e2a-ac50-4b8d7858c91e-openshift-service-ca\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.544022 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.543991 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-openshift-service-ca\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.545670 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.545643 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-maas-api-tls\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.545791 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.545766 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25346f5b-b129-4e2a-ac50-4b8d7858c91e-metrics-certs\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.545898 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.545882 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25346f5b-b129-4e2a-ac50-4b8d7858c91e-maas-api-tls\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.545940 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.545894 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-metrics-certs\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.555657 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.555636 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crh7z\" (UniqueName: \"kubernetes.io/projected/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-kube-api-access-crh7z\") pod \"maas-api-59f797486c-gnhv2\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.555745 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.555666 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmlv\" (UniqueName: \"kubernetes.io/projected/25346f5b-b129-4e2a-ac50-4b8d7858c91e-kube-api-access-hhmlv\") pod \"maas-api-6fcfd65cff-6hdbl\" (UID: \"25346f5b-b129-4e2a-ac50-4b8d7858c91e\") " pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.693698 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.693613 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:13.713649 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.713621 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:13.976266 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:13.976233 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-gnhv2"] Nov 04 00:59:13.978570 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:59:13.978548 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c5ab62_f548_4177_a2f1_e1ccab1a460e.slice/crio-b163186bed575d8b988877b47a22b47acc92f57cc9f4db183f3c984cf10a62ae WatchSource:0}: Error finding container b163186bed575d8b988877b47a22b47acc92f57cc9f4db183f3c984cf10a62ae: Status 404 returned error can't find the container with id b163186bed575d8b988877b47a22b47acc92f57cc9f4db183f3c984cf10a62ae Nov 04 00:59:14.006600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:14.006573 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl"] Nov 04 00:59:14.008643 ip-10-0-138-137 kubenswrapper[6208]: W1104 00:59:14.008613 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25346f5b_b129_4e2a_ac50_4b8d7858c91e.slice/crio-3a99a63b623d43ca47cd26478b0f00caa4ba5c19305dc9f601a94940326cb46e WatchSource:0}: Error finding container 3a99a63b623d43ca47cd26478b0f00caa4ba5c19305dc9f601a94940326cb46e: Status 404 returned error can't find the container with id 3a99a63b623d43ca47cd26478b0f00caa4ba5c19305dc9f601a94940326cb46e Nov 04 00:59:14.090442 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:14.090402 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" event={"ID":"25346f5b-b129-4e2a-ac50-4b8d7858c91e","Type":"ContainerStarted","Data":"3a99a63b623d43ca47cd26478b0f00caa4ba5c19305dc9f601a94940326cb46e"} Nov 04 00:59:14.091721 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:14.091691 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" event={"ID":"d7c5ab62-f548-4177-a2f1-e1ccab1a460e","Type":"ContainerStarted","Data":"b163186bed575d8b988877b47a22b47acc92f57cc9f4db183f3c984cf10a62ae"} Nov 04 00:59:15.098094 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:15.098029 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" event={"ID":"25346f5b-b129-4e2a-ac50-4b8d7858c91e","Type":"ContainerStarted","Data":"9993fc1025248693a6518f9177c125c0cc7418c2096374a9b4a36b5d2762c4ad"} Nov 04 00:59:15.098600 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:15.098121 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:15.099614 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:15.099586 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" event={"ID":"d7c5ab62-f548-4177-a2f1-e1ccab1a460e","Type":"ContainerStarted","Data":"d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5"} Nov 04 00:59:15.099737 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:15.099640 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:15.119582 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:15.119529 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" podStartSLOduration=1.598395874 podStartE2EDuration="2.119517735s" totalImagesPullingTime="521.121861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:59:15.118240876 +0000 UTC m=+5373.940837037" watchObservedRunningTime="2026-11-04 00:59:15.119517735 +0000 UTC m=+5373.942113897" Nov 04 00:59:15.138697 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:15.138650 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" podStartSLOduration=1.618902542 podStartE2EDuration="2.138633753s" totalImagesPullingTime="519.731211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:59:15.138242984 +0000 UTC m=+5373.960839145" watchObservedRunningTime="2026-11-04 00:59:15.138633753 +0000 UTC m=+5373.961229916" Nov 04 00:59:17.447870 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:17.447827 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:20.824283 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:20.824243 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:21.110848 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.110768 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6fcfd65cff-6hdbl" Nov 04 00:59:21.111248 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.111225 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:21.167148 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.167114 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-gnhv2"] Nov 04 00:59:21.167392 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.167325 6208 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" podUID="d7c5ab62-f548-4177-a2f1-e1ccab1a460e" containerName="maas-api" containerID="cri-o://d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5" gracePeriod=30 Nov 04 00:59:21.424298 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.424273 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:21.517142 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.517107 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-openshift-service-ca\") pod \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " Nov 04 00:59:21.517331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.517153 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-maas-api-tls\") pod \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " Nov 04 00:59:21.517331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.517252 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-kube-api-access-crh7z\" (UniqueName: \"kubernetes.io/projected/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-kube-api-access-crh7z\") pod \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " Nov 04 00:59:21.517331 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.517302 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-metrics-certs\") pod \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\" (UID: \"d7c5ab62-f548-4177-a2f1-e1ccab1a460e\") " Nov 04 00:59:21.517544 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.517511 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-openshift-service-ca" pod "d7c5ab62-f548-4177-a2f1-e1ccab1a460e" (UID: "d7c5ab62-f548-4177-a2f1-e1ccab1a460e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:21.519493 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.519462 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-metrics-certs" pod "d7c5ab62-f548-4177-a2f1-e1ccab1a460e" (UID: "d7c5ab62-f548-4177-a2f1-e1ccab1a460e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:59:21.519608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.519498 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-maas-api-tls" pod "d7c5ab62-f548-4177-a2f1-e1ccab1a460e" (UID: "d7c5ab62-f548-4177-a2f1-e1ccab1a460e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:59:21.519608 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.519518 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-kube-api-access-crh7z" pod "d7c5ab62-f548-4177-a2f1-e1ccab1a460e" (UID: "d7c5ab62-f548-4177-a2f1-e1ccab1a460e"). InnerVolumeSpecName "kube-api-access-crh7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:59:21.618871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.618842 6208 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-metrics-certs\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:21.618871 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.618868 6208 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-openshift-service-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:21.619078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.618878 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-maas-api-tls\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:21.619078 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:21.618889 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crh7z\" (UniqueName: \"kubernetes.io/projected/d7c5ab62-f548-4177-a2f1-e1ccab1a460e-kube-api-access-crh7z\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:22.137967 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:22.137928 6208 generic.go:356] "Generic (PLEG): container finished" podID="d7c5ab62-f548-4177-a2f1-e1ccab1a460e" containerID="d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5" exitCode=0 Nov 04 00:59:22.138404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:22.137988 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" Nov 04 00:59:22.138404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:22.138007 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" event={"ID":"d7c5ab62-f548-4177-a2f1-e1ccab1a460e","Type":"ContainerDied","Data":"d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5"} Nov 04 00:59:22.138404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:22.138091 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f797486c-gnhv2" event={"ID":"d7c5ab62-f548-4177-a2f1-e1ccab1a460e","Type":"ContainerDied","Data":"b163186bed575d8b988877b47a22b47acc92f57cc9f4db183f3c984cf10a62ae"} Nov 04 00:59:22.138404 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:22.138124 6208 scope.go:122] "RemoveContainer" containerID="d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5" Nov 04 00:59:22.147899 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:22.147880 6208 scope.go:122] "RemoveContainer" containerID="d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5" Nov 04 00:59:22.148191 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:59:22.148165 6208 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5\": container with ID starting with d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5 not found: ID does not exist" containerID="d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5" Nov 04 00:59:22.148256 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:22.148199 6208 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5"} err="failed to get container status \"d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5\": rpc error: code = NotFound desc = could not find container \"d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5\": container with ID starting with d7bb9bc07b453b87705b7995bafabd60c998774da419193ffbce8e07a50da0e5 not found: ID does not exist" Nov 04 00:59:22.163003 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:22.162981 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-gnhv2"] Nov 04 00:59:22.167264 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:22.167241 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f797486c-gnhv2"] Nov 04 00:59:23.314857 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:23.314812 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c5ab62-f548-4177-a2f1-e1ccab1a460e" path="/var/lib/kubelet/pods/d7c5ab62-f548-4177-a2f1-e1ccab1a460e/volumes" Nov 04 00:59:25.248890 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:25.248847 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 00:59:35.703123 ip-10-0-138-137 kubenswrapper[6208]: E1104 00:59:35.703074 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 00:59:43.336810 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:43.336777 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:59:43.342865 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:43.342842 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 00:59:47.536141 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:47.536107 6208 scope.go:122] "RemoveContainer" containerID="fce657c0514b1dd07917e2e357a57481678b75e20d8e66926c3167d1e30b0625" Nov 04 00:59:47.545550 ip-10-0-138-137 kubenswrapper[6208]: I1104 00:59:47.545527 6208 scope.go:122] "RemoveContainer" containerID="44e0647598d1caf4dd21bcba1dcde6a4d1212d4b980ed01dfe6523d69decbe66" Nov 04 01:00:02.492820 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:02.492775 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:00:02.588909 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:02.588871 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:00:05.752535 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:00:05.752490 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:00:09.532933 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:09.532890 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:00:11.042391 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.042353 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7"] Nov 04 01:00:11.042814 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.042714 6208 cpu_manager.go:414] "containerMap: removing container" podUID="d7c5ab62-f548-4177-a2f1-e1ccab1a460e" containerName="maas-api" Nov 04 01:00:11.042814 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.042725 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c5ab62-f548-4177-a2f1-e1ccab1a460e" containerName="maas-api" Nov 04 01:00:11.042908 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.042815 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c5ab62-f548-4177-a2f1-e1ccab1a460e" containerName="maas-api" Nov 04 01:00:11.046128 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.046098 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.054607 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.054583 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ghz4f\"" Nov 04 01:00:11.057844 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.054784 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:00:11.057844 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.057717 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7"] Nov 04 01:00:11.076324 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.076277 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9cad65-r74k7\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.076324 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.076324 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9cad65-r74k7\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.076651 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.076456 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5r9\" (UniqueName: \"kubernetes.io/projected/fdfaaab9-0635-4002-9b8d-f7e830db5642-kube-api-access-bl5r9\") pod \"maas-api-revoke-sub-be9cad65-r74k7\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.177930 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.177894 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5r9\" (UniqueName: \"kubernetes.io/projected/fdfaaab9-0635-4002-9b8d-f7e830db5642-kube-api-access-bl5r9\") pod \"maas-api-revoke-sub-be9cad65-r74k7\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.178134 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.177963 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9cad65-r74k7\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.178192 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.178145 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9cad65-r74k7\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.178657 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.178637 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9cad65-r74k7\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.180476 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.180458 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9cad65-r74k7\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.204225 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.204184 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5r9\" (UniqueName: \"kubernetes.io/projected/fdfaaab9-0635-4002-9b8d-f7e830db5642-kube-api-access-bl5r9\") pod \"maas-api-revoke-sub-be9cad65-r74k7\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.362804 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.362722 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:11.640874 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:11.640838 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7"] Nov 04 01:00:11.642538 ip-10-0-138-137 kubenswrapper[6208]: W1104 01:00:11.642510 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfaaab9_0635_4002_9b8d_f7e830db5642.slice/crio-68c90f432ce7b6b7ae3df28f89785b8254c65d0195a2685752db7f0db3c873a1 WatchSource:0}: Error finding container 68c90f432ce7b6b7ae3df28f89785b8254c65d0195a2685752db7f0db3c873a1: Status 404 returned error can't find the container with id 68c90f432ce7b6b7ae3df28f89785b8254c65d0195a2685752db7f0db3c873a1 Nov 04 01:00:12.446019 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:12.445926 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" event={"ID":"fdfaaab9-0635-4002-9b8d-f7e830db5642","Type":"ContainerStarted","Data":"7c7e0a1a7d0eeaef70b5d0480b4df31a54ad29ace579e85d36705725a17b1450"} Nov 04 01:00:12.446019 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:12.445985 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" event={"ID":"fdfaaab9-0635-4002-9b8d-f7e830db5642","Type":"ContainerStarted","Data":"68c90f432ce7b6b7ae3df28f89785b8254c65d0195a2685752db7f0db3c873a1"} Nov 04 01:00:12.467117 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:12.467064 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" podStartSLOduration=1.467049517 podStartE2EDuration="1.467049517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:00:12.46467967 +0000 UTC m=+5431.287275847" watchObservedRunningTime="2026-11-04 01:00:12.467049517 +0000 UTC m=+5431.289645669" Nov 04 01:00:13.452490 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:13.452449 6208 generic.go:356] "Generic (PLEG): container finished" podID="fdfaaab9-0635-4002-9b8d-f7e830db5642" containerID="7c7e0a1a7d0eeaef70b5d0480b4df31a54ad29ace579e85d36705725a17b1450" exitCode=0 Nov 04 01:00:13.452887 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:13.452532 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" event={"ID":"fdfaaab9-0635-4002-9b8d-f7e830db5642","Type":"ContainerDied","Data":"7c7e0a1a7d0eeaef70b5d0480b4df31a54ad29ace579e85d36705725a17b1450"} Nov 04 01:00:14.595136 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.595108 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:14.712527 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.712447 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-xks-ca\") pod \"fdfaaab9-0635-4002-9b8d-f7e830db5642\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " Nov 04 01:00:14.712527 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.712500 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-openshift-ca\") pod \"fdfaaab9-0635-4002-9b8d-f7e830db5642\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " Nov 04 01:00:14.712719 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.712567 6208 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfaaab9-0635-4002-9b8d-f7e830db5642-kube-api-access-bl5r9\" (UniqueName: \"kubernetes.io/projected/fdfaaab9-0635-4002-9b8d-f7e830db5642-kube-api-access-bl5r9\") pod \"fdfaaab9-0635-4002-9b8d-f7e830db5642\" (UID: \"fdfaaab9-0635-4002-9b8d-f7e830db5642\") " Nov 04 01:00:14.712915 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.712891 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-openshift-ca" pod "fdfaaab9-0635-4002-9b8d-f7e830db5642" (UID: "fdfaaab9-0635-4002-9b8d-f7e830db5642"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:00:14.714574 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.714542 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-xks-ca" pod "fdfaaab9-0635-4002-9b8d-f7e830db5642" (UID: "fdfaaab9-0635-4002-9b8d-f7e830db5642"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:00:14.714574 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.714558 6208 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfaaab9-0635-4002-9b8d-f7e830db5642-kube-api-access-bl5r9" pod "fdfaaab9-0635-4002-9b8d-f7e830db5642" (UID: "fdfaaab9-0635-4002-9b8d-f7e830db5642"). InnerVolumeSpecName "kube-api-access-bl5r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:00:14.813348 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.813309 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-xks-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:14.813348 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.813340 6208 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfaaab9-0635-4002-9b8d-f7e830db5642-maas-api-openshift-ca\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:14.813348 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:14.813354 6208 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl5r9\" (UniqueName: \"kubernetes.io/projected/fdfaaab9-0635-4002-9b8d-f7e830db5642-kube-api-access-bl5r9\") on node \"ip-10-0-138-137.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:15.465910 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:15.465877 6208 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" Nov 04 01:00:15.466129 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:15.465913 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7" event={"ID":"fdfaaab9-0635-4002-9b8d-f7e830db5642","Type":"ContainerDied","Data":"68c90f432ce7b6b7ae3df28f89785b8254c65d0195a2685752db7f0db3c873a1"} Nov 04 01:00:15.466129 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:15.465959 6208 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68c90f432ce7b6b7ae3df28f89785b8254c65d0195a2685752db7f0db3c873a1" Nov 04 01:00:20.907847 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:20.907809 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:00:35.747795 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:00:35.747743 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:00:47.578693 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:00:47.578651 6208 scope.go:122] "RemoveContainer" containerID="88774b9c5691b55f7d87fd4b9f46c333c6cf65afe16aceb45bf90bfb0463d136" Nov 04 01:01:05.704196 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:01:05.704158 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:01:09.592743 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:01:09.592702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:01:17.548201 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:01:17.548154 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ldsgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:01:17.548712 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:01:17.548218 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ldsgp" podUID="eb0530c0-024b-438e-8cd4-e530338b9744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:01:20.973118 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:01:20.972986 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:01:35.702962 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:01:35.702923 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:01:40.327056 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:01:40.326795 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:00.028573 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:02:00.028527 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6"] Nov 04 01:02:00.030357 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:02:00.030329 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rt2l6"] Nov 04 01:02:01.292470 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:02:01.292437 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24bc803c-8cfa-4de7-95ae-7e33b11faa10" path="/var/lib/kubelet/pods/24bc803c-8cfa-4de7-95ae-7e33b11faa10/volumes" Nov 04 01:02:05.704053 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:02:05.704005 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:02:09.659469 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:02:09.659425 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:15.658193 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:02:15.658148 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:02:15.658824 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:02:15.658789 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:02:21.024231 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:02:21.024192 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:02:35.704014 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:02:35.703975 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:02:40.393580 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:02:40.393541 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:05.703937 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:03:05.703895 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:03:09.708341 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:03:09.708300 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:15.657289 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:03:15.657231 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:03:15.658068 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:03:15.657995 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:03:21.105168 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:03:21.105132 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:35.750949 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:03:35.750905 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:03:46.634441 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:03:46.634399 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:03:47.617780 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:03:47.617740 6208 scope.go:122] "RemoveContainer" containerID="49ee12f902a951c3d5f7a62f2a1883353a91d1811d108c831e056723c05065b6" Nov 04 01:03:48.026322 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:03:48.026282 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn"] Nov 04 01:03:48.028888 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:03:48.028861 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cnwfn"] Nov 04 01:03:49.293592 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:03:49.293555 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a88294-9db4-4ce5-bb00-91f9d8cba085" path="/var/lib/kubelet/pods/67a88294-9db4-4ce5-bb00-91f9d8cba085/volumes" Nov 04 01:04:05.029358 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:05.029308 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd"] Nov 04 01:04:05.032276 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:05.032248 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l8hd"] Nov 04 01:04:05.312110 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:05.311608 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2300905c-5cae-4639-8f34-469c832a18f1" path="/var/lib/kubelet/pods/2300905c-5cae-4639-8f34-469c832a18f1/volumes" Nov 04 01:04:05.752892 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:04:05.752852 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:04:15.661123 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:15.656205 6208 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-ffbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:04:15.661123 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:15.656299 6208 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-ffbsd" podUID="84fcd33b-388e-4e91-a74e-fe88a560c5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:04:21.133116 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:21.133015 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:04:35.704813 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:04:35.704769 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:04:39.645995 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:39.645951 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b594799b-mkv2k_93e48259-862a-4633-b927-84fd5351d25a/authorino/0.log" Nov 04 01:04:43.374428 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:43.374391 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 01:04:43.382749 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:43.382719 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jjsq6_e063a0d5-49bc-49f6-9f30-25ecd024ea4d/extract/0.log" Nov 04 01:04:45.450378 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:45.450334 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:04:47.641631 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:04:47.641591 6208 scope.go:122] "RemoveContainer" containerID="adca671b5422dd8559aa5df8784f51d62528eea525b0e3ce55a24356e5c2562d" Nov 04 01:05:01.733201 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:01.733160 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/1.log" Nov 04 01:05:05.708592 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:05:05.708545 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:05:13.941657 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:13.941621 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/0.log" Nov 04 01:05:14.537187 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:14.537156 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6fcfd65cff-6hdbl_25346f5b-b129-4e2a-ac50-4b8d7858c91e/maas-api/0.log" Nov 04 01:05:14.779662 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:14.779621 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:05:15.040504 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:15.040469 6208 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7"] Nov 04 01:05:15.043026 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:15.043001 6208 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9cad65-r74k7"] Nov 04 01:05:15.293064 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:15.292971 6208 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfaaab9-0635-4002-9b8d-f7e830db5642" path="/var/lib/kubelet/pods/fdfaaab9-0635-4002-9b8d-f7e830db5642/volumes" Nov 04 01:05:21.191865 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:21.191821 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:05:22.787736 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:22.787702 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:05:30.063492 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:30.063456 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b594799b-mkv2k_93e48259-862a-4633-b927-84fd5351d25a/authorino/0.log" Nov 04 01:05:30.682477 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:30.682446 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jc64k_fd69111b-a7ff-4661-8608-4dbc46282b48/kuadrant-console-plugin/0.log" Nov 04 01:05:35.703682 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:05:35.703640 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:05:43.061726 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:43.061688 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-694bd9f98-rl6lx_ed7ca6de-f794-4ea4-9a65-79dbcde9d373/kube-auth-proxy/0.log" Nov 04 01:05:43.247402 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:43.247355 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-5hv2p_de2726d5-efb3-4832-977b-e2315e48a63c/istio-proxy/0.log" Nov 04 01:05:43.646864 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:43.646784 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ffbsd_84fcd33b-388e-4e91-a74e-fe88a560c5d8/istio-proxy/0.log" Nov 04 01:05:44.034112 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:44.034077 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ldsgp_eb0530c0-024b-438e-8cd4-e530338b9744/istio-proxy/0.log" Nov 04 01:05:45.406247 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:45.406210 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-668f485779-rptm9_3f63c561-d450-4ad8-936a-3a035f80035c/payload-pre-processing/0.log" Nov 04 01:05:45.513989 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:45.513959 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:05:45.584407 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:45.584378 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b99c78677-pfh6c_7f1e58d6-90eb-44a5-a09d-2680a93e4e31/payload-processing/0.log" Nov 04 01:05:47.166048 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:47.166015 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-qq6xc_5c51275a-7e33-4a5c-9300-7e5e8d92c294/main/0.log" Nov 04 01:05:47.519778 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:47.519750 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fj7th_ef180863-1732-423f-a932-989ba01cb99c/main/0.log" Nov 04 01:05:47.665328 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:47.665296 6208 scope.go:122] "RemoveContainer" containerID="6138cd8aef94b04dcb4975ca966ed061fc54d9e65a0ca111f21f9dc133be0c24" Nov 04 01:05:47.894129 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:05:47.894001 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-lg6nd_83c22dc8-99b1-4cbd-b15a-c7d48742940a/main/0.log" Nov 04 01:06:05.703828 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:06:05.703709 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:06:14.841112 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:14.841073 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:06:19.541313 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.541275 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:06:19.557554 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.557517 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5lsvx/must-gather-xj27s"] Nov 04 01:06:19.557917 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.557902 6208 cpu_manager.go:414] "containerMap: removing container" podUID="fdfaaab9-0635-4002-9b8d-f7e830db5642" containerName="revoke-keys" Nov 04 01:06:19.557962 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.557919 6208 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfaaab9-0635-4002-9b8d-f7e830db5642" containerName="revoke-keys" Nov 04 01:06:19.558009 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.557999 6208 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfaaab9-0635-4002-9b8d-f7e830db5642" containerName="revoke-keys" Nov 04 01:06:19.561633 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.561611 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5lsvx/must-gather-xj27s" Nov 04 01:06:19.566249 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.566224 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5lsvx\"/\"default-dockercfg-bjk6g\"" Nov 04 01:06:19.573012 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.572994 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5lsvx\"/\"openshift-service-ca.crt\"" Nov 04 01:06:19.573430 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.573413 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5lsvx\"/\"kube-root-ca.crt\"" Nov 04 01:06:19.579766 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.579741 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5lsvx/must-gather-xj27s"] Nov 04 01:06:19.604857 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.604830 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6br9t\" (UniqueName: \"kubernetes.io/projected/c5687bd7-fb03-4a24-a6fe-70d7270efac6-kube-api-access-6br9t\") pod \"must-gather-xj27s\" (UID: \"c5687bd7-fb03-4a24-a6fe-70d7270efac6\") " pod="openshift-must-gather-5lsvx/must-gather-xj27s" Nov 04 01:06:19.604976 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.604867 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c5687bd7-fb03-4a24-a6fe-70d7270efac6-must-gather-output\") pod \"must-gather-xj27s\" (UID: \"c5687bd7-fb03-4a24-a6fe-70d7270efac6\") " pod="openshift-must-gather-5lsvx/must-gather-xj27s" Nov 04 01:06:19.642292 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.642262 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:06:19.705756 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.705718 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6br9t\" (UniqueName: \"kubernetes.io/projected/c5687bd7-fb03-4a24-a6fe-70d7270efac6-kube-api-access-6br9t\") pod \"must-gather-xj27s\" (UID: \"c5687bd7-fb03-4a24-a6fe-70d7270efac6\") " pod="openshift-must-gather-5lsvx/must-gather-xj27s" Nov 04 01:06:19.705949 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.705765 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c5687bd7-fb03-4a24-a6fe-70d7270efac6-must-gather-output\") pod \"must-gather-xj27s\" (UID: \"c5687bd7-fb03-4a24-a6fe-70d7270efac6\") " pod="openshift-must-gather-5lsvx/must-gather-xj27s" Nov 04 01:06:19.706209 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.706185 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c5687bd7-fb03-4a24-a6fe-70d7270efac6-must-gather-output\") pod \"must-gather-xj27s\" (UID: \"c5687bd7-fb03-4a24-a6fe-70d7270efac6\") " pod="openshift-must-gather-5lsvx/must-gather-xj27s" Nov 04 01:06:19.718223 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.718199 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6br9t\" (UniqueName: \"kubernetes.io/projected/c5687bd7-fb03-4a24-a6fe-70d7270efac6-kube-api-access-6br9t\") pod \"must-gather-xj27s\" (UID: \"c5687bd7-fb03-4a24-a6fe-70d7270efac6\") " pod="openshift-must-gather-5lsvx/must-gather-xj27s" Nov 04 01:06:19.875276 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:19.875185 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5lsvx/must-gather-xj27s" Nov 04 01:06:20.154710 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:20.154677 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5lsvx/must-gather-xj27s"] Nov 04 01:06:20.157422 ip-10-0-138-137 kubenswrapper[6208]: W1104 01:06:20.157395 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5687bd7_fb03_4a24_a6fe_70d7270efac6.slice/crio-17ec875ea112773944716f76c5bb1ded40b97d306b3e04861c1c4b255e16bb9f WatchSource:0}: Error finding container 17ec875ea112773944716f76c5bb1ded40b97d306b3e04861c1c4b255e16bb9f: Status 404 returned error can't find the container with id 17ec875ea112773944716f76c5bb1ded40b97d306b3e04861c1c4b255e16bb9f Nov 04 01:06:20.159237 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:20.159217 6208 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:06:20.810564 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:20.810524 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5lsvx/must-gather-xj27s" event={"ID":"c5687bd7-fb03-4a24-a6fe-70d7270efac6","Type":"ContainerStarted","Data":"17ec875ea112773944716f76c5bb1ded40b97d306b3e04861c1c4b255e16bb9f"} Nov 04 01:06:21.249354 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:21.249319 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:06:23.110324 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:23.110211 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p7zhg_9c784845-c509-4b51-86b8-ce9075600208/global-pull-secret-syncer/0.log" Nov 04 01:06:23.371642 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:23.371537 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" Nov 04 01:06:23.552023 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:23.551989 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-137.us-east-2.compute.internal_92a1bdd4dac917849714e10bd3b332b6/haproxy/0.log" Nov 04 01:06:26.850157 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:26.850109 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5lsvx/must-gather-xj27s" event={"ID":"c5687bd7-fb03-4a24-a6fe-70d7270efac6","Type":"ContainerStarted","Data":"52d6f20614f97238b3209c590726aaa31b41b7dc614064f870d5e625143f46e9"} Nov 04 01:06:26.850157 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:26.850162 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5lsvx/must-gather-xj27s" event={"ID":"c5687bd7-fb03-4a24-a6fe-70d7270efac6","Type":"ContainerStarted","Data":"4437bfedc77ebe05bf21576c2cf8eb9fbe1e308341df475098c9b47140ebd9bc"} Nov 04 01:06:26.870759 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:26.870703 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5lsvx/must-gather-xj27s" podStartSLOduration=1.730772186 podStartE2EDuration="7.8706881s" totalImagesPullingTime="6.139915914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:26.869227077 +0000 UTC m=+5805.691823239" watchObservedRunningTime="2026-11-04 01:06:26.8706881 +0000 UTC m=+5805.693284261" Nov 04 01:06:27.530906 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:27.530865 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/1.log" Nov 04 01:06:27.546101 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:27.546068 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/0.log" Nov 04 01:06:30.051861 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:30.051814 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/1.log" Nov 04 01:06:30.065864 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:30.065832 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/0.log" Nov 04 01:06:32.654167 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:32.654131 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/1.log" Nov 04 01:06:32.668922 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:32.668896 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/0.log" Nov 04 01:06:35.708738 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:06:35.708693 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:06:40.961007 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:40.960974 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-694bd9f98-rl6lx_ed7ca6de-f794-4ea4-9a65-79dbcde9d373/kube-auth-proxy/0.log" Nov 04 01:06:40.982718 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:40.982687 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-5hv2p_de2726d5-efb3-4832-977b-e2315e48a63c/istio-proxy/0.log" Nov 04 01:06:41.015859 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:41.015825 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ffbsd_84fcd33b-388e-4e91-a74e-fe88a560c5d8/istio-proxy/0.log" Nov 04 01:06:41.059752 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:41.059713 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ldsgp_eb0530c0-024b-438e-8cd4-e530338b9744/istio-proxy/0.log" Nov 04 01:06:41.168880 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:41.168839 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-668f485779-rptm9_3f63c561-d450-4ad8-936a-3a035f80035c/payload-pre-processing/0.log" Nov 04 01:06:41.182587 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:41.182548 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b99c78677-pfh6c_7f1e58d6-90eb-44a5-a09d-2680a93e4e31/payload-processing/0.log" Nov 04 01:06:44.674226 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:44.674187 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b594799b-mkv2k_93e48259-862a-4633-b927-84fd5351d25a/authorino/0.log" Nov 04 01:06:44.732518 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:44.732491 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jc64k_fd69111b-a7ff-4661-8608-4dbc46282b48/kuadrant-console-plugin/0.log" Nov 04 01:06:45.622850 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:45.622801 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:06:47.687207 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:47.687169 6208 scope.go:122] "RemoveContainer" containerID="7c7e0a1a7d0eeaef70b5d0480b4df31a54ad29ace579e85d36705725a17b1450" Nov 04 01:06:49.178829 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:49.178791 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b594799b-mkv2k_93e48259-862a-4633-b927-84fd5351d25a/authorino/0.log" Nov 04 01:06:49.630839 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:49.630807 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jc64k_fd69111b-a7ff-4661-8608-4dbc46282b48/kuadrant-console-plugin/0.log" Nov 04 01:06:51.351815 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.351761 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsslm_fdc30eef-1cd6-449c-9aef-d15732f1a2f8/node-exporter/0.log" Nov 04 01:06:51.380230 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.380194 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsslm_fdc30eef-1cd6-449c-9aef-d15732f1a2f8/kube-rbac-proxy/0.log" Nov 04 01:06:51.408833 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.408807 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dsslm_fdc30eef-1cd6-449c-9aef-d15732f1a2f8/init-textfile/0.log" Nov 04 01:06:51.550794 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.550765 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e831b41a-40d4-4abe-a71c-f30d340c169d/prometheus/0.log" Nov 04 01:06:51.577755 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.577726 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e831b41a-40d4-4abe-a71c-f30d340c169d/config-reloader/0.log" Nov 04 01:06:51.608879 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.608805 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e831b41a-40d4-4abe-a71c-f30d340c169d/thanos-sidecar/0.log" Nov 04 01:06:51.638398 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.638372 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e831b41a-40d4-4abe-a71c-f30d340c169d/kube-rbac-proxy-web/0.log" Nov 04 01:06:51.667284 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.667259 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e831b41a-40d4-4abe-a71c-f30d340c169d/kube-rbac-proxy/0.log" Nov 04 01:06:51.707140 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.707111 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e831b41a-40d4-4abe-a71c-f30d340c169d/kube-rbac-proxy-thanos/0.log" Nov 04 01:06:51.738509 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.738481 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e831b41a-40d4-4abe-a71c-f30d340c169d/init-config-reloader/0.log" Nov 04 01:06:51.784069 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.784020 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-fzfcl_37f77e6f-4bd8-4a9e-ab23-c1337365cac1/prometheus-operator/0.log" Nov 04 01:06:51.809770 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.809738 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-fzfcl_37f77e6f-4bd8-4a9e-ab23-c1337365cac1/kube-rbac-proxy/0.log" Nov 04 01:06:51.888963 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.888884 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df744b67-frfj2_3a95c441-120e-44ce-8b68-516030f94a91/telemeter-client/0.log" Nov 04 01:06:51.916550 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.916521 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df744b67-frfj2_3a95c441-120e-44ce-8b68-516030f94a91/reload/0.log" Nov 04 01:06:51.945668 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:51.945642 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5df744b67-frfj2_3a95c441-120e-44ce-8b68-516030f94a91/kube-rbac-proxy/0.log" Nov 04 01:06:54.251815 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:54.251780 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-86b9b49bc8-j9wrj_2378e029-438f-4f14-93b2-5b65a289eb40/console/0.log" Nov 04 01:06:54.949994 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:54.949951 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-597b95cb84-mcznz_42b1afcd-8bf2-49ff-b50d-5cccb24ae831/cert-manager-controller/0.log" Nov 04 01:06:54.964791 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:54.964736 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-58hps_06b6b9d6-7ac8-4541-936f-338efb90765f/cert-manager-cainjector/0.log" Nov 04 01:06:54.980131 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:54.980104 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-c5hhg_63ac6df1-998d-43e6-bf49-2aeff97e4a4b/cert-manager-webhook/0.log" Nov 04 01:06:55.559340 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:55.559291 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-qc9b7_e27fd8a4-1210-4b3a-8787-63ac8da2a2ad/manager/0.log" Nov 04 01:06:55.566353 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:55.566330 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-qc9b7_e27fd8a4-1210-4b3a-8787-63ac8da2a2ad/copy-manifests/0.log" Nov 04 01:06:56.045736 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:56.045560 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h4spw_2bc833b9-89ed-4f1a-8dec-602fffd9c072/dns/0.log" Nov 04 01:06:56.072770 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:56.072735 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h4spw_2bc833b9-89ed-4f1a-8dec-602fffd9c072/kube-rbac-proxy/0.log" Nov 04 01:06:56.183059 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:56.182917 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/1.log" Nov 04 01:06:56.197337 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:56.197309 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/0.log" Nov 04 01:06:56.226958 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:56.226916 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vsrwg_1855ee11-b7e8-49f7-bf9d-4439ac060f20/dns-node-resolver/0.log" Nov 04 01:06:57.065235 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:06:57.065201 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dbjll_58a707ad-9256-4eb1-bf0f-eae7ab6c8d32/node-ca/0.log" Nov 04 01:07:05.705726 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:07:05.705680 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:07:06.187765 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.187659 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-694bd9f98-rl6lx_ed7ca6de-f794-4ea4-9a65-79dbcde9d373/kube-auth-proxy/0.log" Nov 04 01:07:06.230531 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.230498 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-5hv2p_de2726d5-efb3-4832-977b-e2315e48a63c/istio-proxy/0.log" Nov 04 01:07:06.302238 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.302196 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ffbsd_84fcd33b-388e-4e91-a74e-fe88a560c5d8/istio-proxy/0.log" Nov 04 01:07:06.369746 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.369707 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ldsgp_eb0530c0-024b-438e-8cd4-e530338b9744/istio-proxy/0.log" Nov 04 01:07:06.508397 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.508366 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-694bd9f98-rl6lx_ed7ca6de-f794-4ea4-9a65-79dbcde9d373/kube-auth-proxy/0.log" Nov 04 01:07:06.531643 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.531596 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-5hv2p_de2726d5-efb3-4832-977b-e2315e48a63c/istio-proxy/0.log" Nov 04 01:07:06.571138 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.571097 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ffbsd_84fcd33b-388e-4e91-a74e-fe88a560c5d8/istio-proxy/0.log" Nov 04 01:07:06.604668 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.604626 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-ldsgp_eb0530c0-024b-438e-8cd4-e530338b9744/istio-proxy/0.log" Nov 04 01:07:06.608997 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.608965 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-668f485779-rptm9_3f63c561-d450-4ad8-936a-3a035f80035c/payload-pre-processing/0.log" Nov 04 01:07:06.642137 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.642097 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b99c78677-pfh6c_7f1e58d6-90eb-44a5-a09d-2680a93e4e31/payload-processing/0.log" Nov 04 01:07:06.713573 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.713542 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-668f485779-rptm9_3f63c561-d450-4ad8-936a-3a035f80035c/payload-pre-processing/0.log" Nov 04 01:07:06.727158 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:06.727132 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b99c78677-pfh6c_7f1e58d6-90eb-44a5-a09d-2680a93e4e31/payload-processing/0.log" Nov 04 01:07:08.970053 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:08.970010 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-74c78cbdbf-gxkf7_07f6fefb-f056-4495-a778-187e86acc017/manager/1.log" Nov 04 01:07:08.970666 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:08.970641 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-74c78cbdbf-gxkf7_07f6fefb-f056-4495-a778-187e86acc017/manager/0.log" Nov 04 01:07:08.990755 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:08.990730 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pqtnl_17b0bd49-65e6-4393-88ba-b578c5eab313/serve-healthcheck-canary/0.log" Nov 04 01:07:09.011384 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:09.011355 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-nngcn_50e5736e-cc05-4ce1-801b-570844a9a436/jobset-operator/0.log" Nov 04 01:07:09.582838 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:09.582805 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qpvg_9fc8829d-b25c-4952-8cd8-816aa35089ac/kube-rbac-proxy/0.log" Nov 04 01:07:09.611336 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:09.611306 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qpvg_9fc8829d-b25c-4952-8cd8-816aa35089ac/exporter/0.log" Nov 04 01:07:09.645625 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:09.645597 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qpvg_9fc8829d-b25c-4952-8cd8-816aa35089ac/extractor/0.log" Nov 04 01:07:11.397505 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.397463 6208 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5"] Nov 04 01:07:11.404726 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.404696 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.409909 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.409872 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5"] Nov 04 01:07:11.499284 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.499235 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-595p9\"/\"kube-root-ca.crt\"" Nov 04 01:07:11.499284 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.499250 6208 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-595p9\"/\"openshift-service-ca.crt\"" Nov 04 01:07:11.499558 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.499375 6208 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-595p9\"/\"default-dockercfg-whg7p\"" Nov 04 01:07:11.542838 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.542800 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zjw\" (UniqueName: \"kubernetes.io/projected/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-kube-api-access-g9zjw\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.543026 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.542927 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-sys\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.543026 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.542981 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-proc\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.543150 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.543089 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-podres\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.543150 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.543128 6208 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-lib-modules\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.643799 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.643750 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zjw\" (UniqueName: \"kubernetes.io/projected/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-kube-api-access-g9zjw\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.643999 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.643829 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-sys\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.643999 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.643858 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-proc\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.643999 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.643941 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-podres\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.644203 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.644001 6208 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-lib-modules\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.644203 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.644012 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-sys\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.644203 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.644071 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-proc\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.644203 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.644167 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-lib-modules\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.644203 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.644179 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-podres\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.657170 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.657112 6208 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zjw\" (UniqueName: \"kubernetes.io/projected/437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34-kube-api-access-g9zjw\") pod \"perf-node-gather-daemonset-sbwb5\" (UID: \"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34\") " pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:11.727876 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:11.727838 6208 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:12.009293 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:12.009084 6208 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5"] Nov 04 01:07:12.013395 ip-10-0-138-137 kubenswrapper[6208]: W1104 01:07:12.013362 6208 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod437ad7fe_fa02_4f76_9f44_4b2f1c8f6c34.slice/crio-d948c94da840114b12a7f96dcc2161109da6bee47d00012fa4cc3fcfe6a90aad WatchSource:0}: Error finding container d948c94da840114b12a7f96dcc2161109da6bee47d00012fa4cc3fcfe6a90aad: Status 404 returned error can't find the container with id d948c94da840114b12a7f96dcc2161109da6bee47d00012fa4cc3fcfe6a90aad Nov 04 01:07:12.136105 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:12.136064 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" event={"ID":"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34","Type":"ContainerStarted","Data":"d948c94da840114b12a7f96dcc2161109da6bee47d00012fa4cc3fcfe6a90aad"} Nov 04 01:07:12.309225 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:12.309130 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-qc9b7_e27fd8a4-1210-4b3a-8787-63ac8da2a2ad/manager/0.log" Nov 04 01:07:12.333257 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:12.333232 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-775bfbdf99-qc9b7_e27fd8a4-1210-4b3a-8787-63ac8da2a2ad/copy-manifests/0.log" Nov 04 01:07:13.142313 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:13.142273 6208 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" event={"ID":"437ad7fe-fa02-4f76-9f44-4b2f1c8f6c34","Type":"ContainerStarted","Data":"f2fe2dabfa5a4c48a3091571899f2bf629705c23879ea77a2abca77ce1481c49"} Nov 04 01:07:13.142758 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:13.142374 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:13.161114 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:13.161066 6208 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" podStartSLOduration=2.16105263 podStartE2EDuration="2.16105263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:07:13.160494615 +0000 UTC m=+5851.983090778" watchObservedRunningTime="2026-11-04 01:07:13.16105263 +0000 UTC m=+5851.983648792" Nov 04 01:07:13.507160 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:13.507120 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-74c78cbdbf-gxkf7_07f6fefb-f056-4495-a778-187e86acc017/manager/1.log" Nov 04 01:07:13.507358 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:13.507340 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-74c78cbdbf-gxkf7_07f6fefb-f056-4495-a778-187e86acc017/manager/0.log" Nov 04 01:07:13.536004 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:13.535972 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-nngcn_50e5736e-cc05-4ce1-801b-570844a9a436/jobset-operator/0.log" Nov 04 01:07:14.254707 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:14.252840 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/1.log" Nov 04 01:07:14.276066 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:14.274864 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6db599696d-4njr8_c9bf53bb-bfda-4935-8536-dd79e6e7c6c7/manager/0.log" Nov 04 01:07:14.929873 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:14.926148 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:07:19.166080 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:19.166026 6208 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-595p9/perf-node-gather-daemonset-sbwb5" Nov 04 01:07:19.852224 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:19.852126 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b594799b-mkv2k_93e48259-862a-4633-b927-84fd5351d25a/authorino/0.log" Nov 04 01:07:19.908375 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:19.908346 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jc64k_fd69111b-a7ff-4661-8608-4dbc46282b48/kuadrant-console-plugin/0.log" Nov 04 01:07:21.317664 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:21.317574 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:07:23.946114 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:23.946078 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nsk4l_271a3854-3368-425b-9fcd-690738740bb8/kube-multus-additional-cni-plugins/0.log" Nov 04 01:07:23.983149 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:23.983114 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nsk4l_271a3854-3368-425b-9fcd-690738740bb8/egress-router-binary-copy/0.log" Nov 04 01:07:24.015886 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:24.015850 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nsk4l_271a3854-3368-425b-9fcd-690738740bb8/cni-plugins/0.log" Nov 04 01:07:24.057452 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:24.057416 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nsk4l_271a3854-3368-425b-9fcd-690738740bb8/bond-cni-plugin/0.log" Nov 04 01:07:24.087423 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:24.087386 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nsk4l_271a3854-3368-425b-9fcd-690738740bb8/routeoverride-cni/0.log" Nov 04 01:07:24.123276 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:24.123243 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nsk4l_271a3854-3368-425b-9fcd-690738740bb8/whereabouts-cni-bincopy/0.log" Nov 04 01:07:24.155138 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:24.155108 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nsk4l_271a3854-3368-425b-9fcd-690738740bb8/whereabouts-cni/0.log" Nov 04 01:07:24.642708 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:24.642668 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z2zls_36489906-6e8f-4aa6-ae96-f872732808ed/kube-multus/0.log" Nov 04 01:07:24.765781 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:24.765746 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-chg92_f42ad3cc-de80-43bb-b548-dfa6c78618e5/network-metrics-daemon/0.log" Nov 04 01:07:24.788008 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:24.787974 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-chg92_f42ad3cc-de80-43bb-b548-dfa6c78618e5/kube-rbac-proxy/0.log" Nov 04 01:07:25.916632 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:25.916588 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-86q5p_fecf1197-13f8-4c4a-a878-c784e18385d6/ovn-controller/0.log" Nov 04 01:07:25.939790 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:25.939746 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-86q5p_fecf1197-13f8-4c4a-a878-c784e18385d6/ovn-acl-logging/0.log" Nov 04 01:07:25.990284 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:25.990250 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-86q5p_fecf1197-13f8-4c4a-a878-c784e18385d6/kube-rbac-proxy-node/0.log" Nov 04 01:07:26.019482 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:26.019454 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-86q5p_fecf1197-13f8-4c4a-a878-c784e18385d6/kube-rbac-proxy-ovn-metrics/0.log" Nov 04 01:07:26.047797 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:26.047764 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-86q5p_fecf1197-13f8-4c4a-a878-c784e18385d6/northd/0.log" Nov 04 01:07:26.089503 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:26.089466 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-86q5p_fecf1197-13f8-4c4a-a878-c784e18385d6/nbdb/0.log" Nov 04 01:07:26.119875 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:26.119834 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-86q5p_fecf1197-13f8-4c4a-a878-c784e18385d6/sbdb/0.log" Nov 04 01:07:26.456432 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:26.456395 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-86q5p_fecf1197-13f8-4c4a-a878-c784e18385d6/ovnkube-controller/0.log" Nov 04 01:07:28.895800 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:28.895763 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-s7hm2_72d93d4f-2af9-44d5-b8e5-542b30760f34/network-check-target-container/0.log" Nov 04 01:07:30.343704 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:30.343656 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-msjvc_dc219915-7ac3-4e9b-863f-4c641a8b6540/iptables-alerter/0.log" Nov 04 01:07:31.275284 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:31.275249 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-5ss7r_34cd9f7a-0794-465b-9443-d6723c5505fb/tuned/0.log" Nov 04 01:07:34.842001 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:34.841958 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:07:35.691017 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:35.690970 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:07:35.710543 ip-10-0-138-137 kubenswrapper[6208]: E1104 01:07:35.710501 6208 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qpvg" containerName="extractor" Nov 04 01:07:36.009239 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:36.009203 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xs64r_ceed0dfc-9a74-469f-a91d-aab8c3fa402d/csi-driver/0.log" Nov 04 01:07:36.040116 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:36.040076 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xs64r_ceed0dfc-9a74-469f-a91d-aab8c3fa402d/csi-node-driver-registrar/0.log" Nov 04 01:07:36.072089 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:36.072059 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-xs64r_ceed0dfc-9a74-469f-a91d-aab8c3fa402d/csi-liveness-probe/0.log" Nov 04 01:07:45.683603 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:45.683562 6208 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ch5g_1e545f4e-ce4d-4a60-ba33-74207fb40c48/konnectivity-agent/0.log" limit=1024 Nov 04 01:07:50.535915 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:50.535874 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55b594799b-mkv2k_93e48259-862a-4633-b927-84fd5351d25a/authorino/0.log" Nov 04 01:07:50.595766 ip-10-0-138-137 kubenswrapper[6208]: I1104 01:07:50.595734 6208 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jc64k_fd69111b-a7ff-4661-8608-4dbc46282b48/kuadrant-console-plugin/0.log"